-- Logs begin at Tue 2022-07-05 09:25:06 UTC, end at Tue 2022-07-05 10:38:33 UTC. -- Jul 05 09:56:21.521605 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 05 09:56:21.530696 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Jul 5 09:56:21 2022] *** Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: nodename: dv-cinder-master-3-iscsi Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: machine: x86_64 Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: clock source: unix Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: pcre jit disabled Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: detected number of CPU cores: 4 Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: current working directory: / Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: detected binary path: /usr/bin/uwsgi-core Jul 05 09:56:21.532766 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: your processes number limit is 47680 Jul 05 09:56:21.533350 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: your memory page size is 4096 bytes Jul 05 09:56:21.533350 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: detected max file descriptor number: 2048 Jul 05 09:56:21.533350 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: lock engine: pthread robust mutexes Jul 05 09:56:21.533350 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: thunder lock: enabled Jul 05 09:56:21.533350 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 05 09:56:21.533350 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] Jul 05 09:56:21.576890 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: Python main interpreter initialized at 0x5571e758a9c0 Jul 05 09:56:21.576890 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: python threads support enabled Jul 05 09:56:21.576890 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: your server socket listen backlog is limited to 100 connections Jul 05 09:56:21.576890 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: your mercy for graceful operations on workers is 90 seconds Jul 05 09:56:21.577246 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: mapped 403077 bytes (393 KB) for 2 cores Jul 05 09:56:21.577315 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: *** Operational MODE: preforking *** Jul 05 09:56:21.577579 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: *** uWSGI is running in multiple interpreter mode *** Jul 05 09:56:21.577579 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: spawned uWSGI master process (pid: 80257) Jul 05 09:56:21.577781 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 05 09:56:21.581709 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: spawned uWSGI worker 1 (pid: 80268, cores: 1) Jul 05 09:56:21.581709 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: spawned uWSGI worker 2 (pid: 80269, cores: 1) Jul 05 09:56:21.581709 dv-cinder-master-3-iscsi devstack@keystone.service[80257]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 05 09:56:27.009985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 05 09:56:27.009985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: from cryptography.utils import int_from_bytes Jul 05 09:56:27.010522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 05 09:56:27.010522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: from cryptography.utils import int_from_bytes Jul 05 09:56:27.010522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 05 09:56:27.010522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: from cryptography.utils import int_from_bytes Jul 05 09:56:27.011051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 05 09:56:27.011051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: from cryptography.utils import int_from_bytes Jul 05 09:56:27.059846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 05 09:56:27.059985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 05 09:56:27.060061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 05 09:56:27.060142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 05 09:56:27.060253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] command line args: [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 05 09:56:27.060406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] command line args: [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 05 09:56:27.060550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 05 09:56:27.060641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 05 09:56:27.060713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ================================================================================ {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 05 09:56:27.060782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ================================================================================ {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 05 09:56:27.060961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] admin_token = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] admin_token = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] config_dir = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] config_dir = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] config_source = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] config_source = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] debug = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] debug = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.061950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.062973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] list_limit = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] list_limit = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] log_config_append = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] log_config_append = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] log_dir = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] log_dir = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] log_file = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.063983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] log_file = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.064981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.065970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.066926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.067046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.067161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.067285 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.067396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.067526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.067789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.20/identity {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.067911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.20/identity {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] publish_errors = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] publish_errors = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.068880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069134 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] standard_threads = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] standard_threads = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.069993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] use_journal = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] use_journal = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] use_json = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] use_json = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.070937 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] use_stderr = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.071061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] use_stderr = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.071182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] use_syslog = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.071297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] use_syslog = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.071415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.071535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 05 09:56:27.071649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.071891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.072947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.073996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.074931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.075927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.076911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.077959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.078904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.079964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080800 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.080991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.081916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.082978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.083978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.084955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.085896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.086972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.087892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.088950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.089877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.090487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.090611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.090727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.090841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.090959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.091898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.092991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.093954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.094905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095256 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.095953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] auth.external = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.096905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] auth.external = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] auth.password = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.097887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] auth.password = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] auth.token = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] auth.token = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.098950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.099883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] credential.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] credential.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.100888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.101940 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.102994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.103943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.104898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.105014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.105129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.105243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.105366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.105483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.105706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.106363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.106488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.106615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.106745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.106864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.106980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.107982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.108989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.109920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.110991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.111931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.112880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.113957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.114888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.115951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.116950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.117995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.118936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.119838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.120995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.121962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.122936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.123887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.124102 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.124273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.124476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.124598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.124715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.124830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.124945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.125888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.126939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.127997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.128952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.129880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] role.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] role.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.130884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] role.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] role.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.131933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.132926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.133971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.134995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.135893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.136008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.136123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.136239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.136362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.136497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.136613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.136731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.137385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.137520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.137638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.137754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.137868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.137983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.138927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.139290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.139407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.139522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.139636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.139752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.139875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.139992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.140956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141071 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.141911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.142965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.connection = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.143934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.connection = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.144952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.145883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.146945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.147995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.148118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.148235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.148349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80268) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 05 09:56:27.148479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 05 09:56:27.148479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80269) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 05 09:56:27.176514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.176690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.176775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.176919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.177001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.177185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.185340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.185646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.213802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.214030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.214146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.214378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.356267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.358601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.405966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.406205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.406427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.406666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.472043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.472276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.472526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.472638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.472749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.472831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.472968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.473048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.473185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.473278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.473404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.473620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.497995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.498226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.498325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.498451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.498563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.498673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.498792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.498883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.499017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.499099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.499238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.499317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.499457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.499675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.510472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.510567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.524944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.525028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.525160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.525241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.525376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80268) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.525464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80269) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 05 09:56:27.543224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.543436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 05 09:56:27.543752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.543834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.543964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 05 09:56:27.544046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 05 09:56:27.544231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.544309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.544516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 05 09:56:27.544609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.544775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.544855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.545032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.545111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.545288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.545543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.555850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.556126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.556347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.556444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.556654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.556736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.556925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.557209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.565503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.565965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.568796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.569078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.569244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.569361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.569524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.569644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.569807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.569927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.570087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.570189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.570366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.570446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.570622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.570723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.570888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.571004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.571148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.571267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.571412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.571532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.571681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.571947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.598365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.598990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.599968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.600246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.600613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.600890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.605514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.606325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.606687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.606783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.606976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.607258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.614012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.614293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.614646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.614928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.618758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.619049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.619332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.619413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.619617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.619699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.619895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.619975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.620173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.620254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.620468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.620551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.620743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.620823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.621011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.621304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.639551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.639827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.640076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.640350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.644398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.644910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.648015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.648307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.648524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.648819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.653961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.654466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.658074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.661247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.661548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.661703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.661835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.662002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.662127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.662298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.662418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.662594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.662697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.662876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.663154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.682228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.682511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.682621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.682776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.682903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.683040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.683166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.683428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.690933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.691338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.692652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.692927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.693067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.693341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.696089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.696369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.696531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.696673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.696810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.696955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.697087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.697286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.697368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.697677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.710486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.710768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.710853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.711049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.711132 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.711331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.711414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.711612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.711695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.711926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.712220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.712539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.713152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.713445 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.732288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.732588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.735507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.735830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.738816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.739143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.740576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.740769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.742201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.742481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.742563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.742765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.742854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.743051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.750135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.750358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.751746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.751956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.754978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.755177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.755259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.755464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.755647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.755741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.755941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.756025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.765274 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.765482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.765567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.765767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.765849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.766043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.773675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80269) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.773887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80268) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 05 09:56:27.773984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.774168 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.774251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.774454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.774536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.774729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.774820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775082 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.775980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.776164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.776466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.776551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.776735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.776829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.777000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.777081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.777267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.777348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80269) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.778062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80268) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 05 09:56:27.816123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x5571e758a9c0 pid: 80268 (default app) Jul 05 09:56:27.816959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x5571e758a9c0 pid: 80269 (default app) Jul 05 09:56:27.819514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24071aa8-64a7-4ea8-ac65-0a930cc92a31 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:27.819810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24071aa8-64a7-4ea8-ac65-0a930cc92a31 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:27.820088 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24071aa8-64a7-4ea8-ac65-0a930cc92a31 None None] PATH_INFO: `/v3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:27.824365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1/1] 10.222.0.20 () {50 vars in 791 bytes} [Tue Jul 5 09:56:27 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:35.423074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b321920-ea43-4a52-b577-45f7cd4fd5ef None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:35.423631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b321920-ea43-4a52-b577-45f7cd4fd5ef None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:35.423631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b321920-ea43-4a52-b577-45f7cd4fd5ef None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:35.427991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1/2] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:35 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:56:35.433158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:35.434037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:35.434224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:35.502050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: INFO dbcounter [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] Registered counter for database keystone Jul 05 09:56:35.523494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG oslo_db.sqlalchemy.engines [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80268) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 05 09:56:35.529674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writer thread running {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 05 09:56:35.707046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG passlib.handlers.bcrypt [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80268) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 05 09:56:35.707471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG passlib.handlers.bcrypt [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80268) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 05 09:56:35.761538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:35.763784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:35.886652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 2/3] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:35 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 454 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Jul 05 09:56:36.069931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b321920-ea43-4a52-b577-45f7cd4fd5ef None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:36.139423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: INFO dbcounter [None req-9b321920-ea43-4a52-b577-45f7cd4fd5ef None None] Registered counter for database keystone Jul 05 09:56:36.158689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG oslo_db.sqlalchemy.engines [None req-9b321920-ea43-4a52-b577-45f7cd4fd5ef None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80269) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 05 09:56:36.164763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writer thread running {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 05 09:56:36.282441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b321920-ea43-4a52-b577-45f7cd4fd5ef None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:36.286679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7988b50-3d5c-4e27-8790-128f8f778c05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:36.287338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7988b50-3d5c-4e27-8790-128f8f778c05 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:36.287622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7988b50-3d5c-4e27-8790-128f8f778c05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:36.287898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7988b50-3d5c-4e27-8790-128f8f778c05 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:36.296013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7988b50-3d5c-4e27-8790-128f8f778c05 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:36.347021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING keystone.common.rbac_enforcer.enforcer [None req-d7988b50-3d5c-4e27-8790-128f8f778c05 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 05 09:56:36.347311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7988b50-3d5c-4e27-8790-128f8f778c05 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:36.356721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 2/4] 10.222.0.20 () {58 vars in 1324 bytes} [Tue Jul 5 09:56:36 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 09:56:36.362779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:36.377599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74980e9a-8042-4a4b-b798-b9e6f1f0a928 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:36.381971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be44c72-c9d1-45d6-998e-3ad60c85c9b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:36.383120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be44c72-c9d1-45d6-998e-3ad60c85c9b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:36.383406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be44c72-c9d1-45d6-998e-3ad60c85c9b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:36.383684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be44c72-c9d1-45d6-998e-3ad60c85c9b9 admin admin] PATH_INFO: `/v3/projects/ba1e700aae1b46039e859b8f609bb931` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:36.385184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be44c72-c9d1-45d6-998e-3ad60c85c9b9 admin admin] RBAC: Authorizing `identity:get_project(project_id=ba1e700aae1b46039e859b8f609bb931)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:36.437634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING keystone.common.rbac_enforcer.enforcer [None req-2be44c72-c9d1-45d6-998e-3ad60c85c9b9 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 05 09:56:36.438145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be44c72-c9d1-45d6-998e-3ad60c85c9b9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:36.442576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 3/5] 10.222.0.20 () {56 vars in 1207 bytes} [Tue Jul 5 09:56:36 2022] GET /identity/v3/projects/ba1e700aae1b46039e859b8f609bb931 => generated 344 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:38.422212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349e7076-f8bb-4929-af80-27ee65935905 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:38.422755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349e7076-f8bb-4929-af80-27ee65935905 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:38.422755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349e7076-f8bb-4929-af80-27ee65935905 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:38.426935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 3/6] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:38.432231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383491dc-cc6a-4d42-8eb8-5d9517fde35f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:38.432675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383491dc-cc6a-4d42-8eb8-5d9517fde35f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:38.432827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383491dc-cc6a-4d42-8eb8-5d9517fde35f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:38.452348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-383491dc-cc6a-4d42-8eb8-5d9517fde35f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:38.453905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-383491dc-cc6a-4d42-8eb8-5d9517fde35f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:38.461684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 4/7] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:38 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 05 09:56:38.483322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349e7076-f8bb-4929-af80-27ee65935905 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:38.495121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349e7076-f8bb-4929-af80-27ee65935905 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:38.499736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49070fd4-3d49-4e8b-8a16-2b116bcf230c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:38.500365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49070fd4-3d49-4e8b-8a16-2b116bcf230c admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:38.500686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49070fd4-3d49-4e8b-8a16-2b116bcf230c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:38.500969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49070fd4-3d49-4e8b-8a16-2b116bcf230c admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:38.508600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49070fd4-3d49-4e8b-8a16-2b116bcf230c admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:38.510370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49070fd4-3d49-4e8b-8a16-2b116bcf230c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:38.519912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 4/8] 10.222.0.20 () {58 vars in 1324 bytes} [Tue Jul 5 09:56:38 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 09:56:38.524948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383491dc-cc6a-4d42-8eb8-5d9517fde35f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:38.533795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383491dc-cc6a-4d42-8eb8-5d9517fde35f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:38.538139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3135a8a-9bee-4118-ae5f-5efb67f53e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:38.538987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3135a8a-9bee-4118-ae5f-5efb67f53e86 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:38.539275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3135a8a-9bee-4118-ae5f-5efb67f53e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:38.539553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3135a8a-9bee-4118-ae5f-5efb67f53e86 admin admin] PATH_INFO: `/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:38.540975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3135a8a-9bee-4118-ae5f-5efb67f53e86 admin admin] RBAC: Authorizing `identity:get_user(user_id=bb9174aa3ac54d708fcd3d9dc7b9b827)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:38.542803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3135a8a-9bee-4118-ae5f-5efb67f53e86 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:38.544355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 5/9] 10.222.0.20 () {56 vars in 1198 bytes} [Tue Jul 5 09:56:38 2022] GET /identity/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:40.605699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bd7ef3-f31c-4853-9bad-1b678f5ddea9 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:40.606209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bd7ef3-f31c-4853-9bad-1b678f5ddea9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:40.606290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bd7ef3-f31c-4853-9bad-1b678f5ddea9 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:40.607615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 5/10] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:40.612982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c7737-7547-4724-a5e1-0cf3bb8d44a3 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:40.613387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c7737-7547-4724-a5e1-0cf3bb8d44a3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:40.613551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89c7737-7547-4724-a5e1-0cf3bb8d44a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:40.614157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463023f5-928f-4e94-b6e2-5f4a52653149 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:40.614454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463023f5-928f-4e94-b6e2-5f4a52653149 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:40.614743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463023f5-928f-4e94-b6e2-5f4a52653149 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:40.616015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 6/11] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:56:40.620907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:40.621710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:40.621995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:40.630798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c89c7737-7547-4724-a5e1-0cf3bb8d44a3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:40.631980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c89c7737-7547-4724-a5e1-0cf3bb8d44a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:40.661557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG passlib.handlers.bcrypt [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80269) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 05 09:56:40.661993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG passlib.handlers.bcrypt [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80269) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 05 09:56:40.709389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:40.710685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 6/12] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:40.711018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:40.733372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89c7737-7547-4724-a5e1-0cf3bb8d44a3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:40.757136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:40.816305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 7/13] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:40.835859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89c7737-7547-4724-a5e1-0cf3bb8d44a3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:40.837376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:40.869677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:40.870412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:40.870698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:40.871572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:40.882956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:40.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:40.895859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:56:40.896738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 7/14] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:56:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:56:40.901768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:40.937950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af20c23a-aee0-42a9-974e-b8974808e0ed None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:40.940211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:40.975860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:40.976802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:40.977101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:40.977384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:40.983906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:41.001703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51aa4209-5a4e-4439-b455-625cd5732e2d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:41.035573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:41.036242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:41.036571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:41.036858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:41.037621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:41.037928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:41.039342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:41.046036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:41.047552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:41.051011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 8/15] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:41.056642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:41.144457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 8/16] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:56:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:56:41.149501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:41.152108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:41.156458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c789b9-eabb-4ae8-96c0-db4c667874d5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:41.189755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:41.190632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:41.190919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:41.191200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] PATH_INFO: `/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:41.192428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] RBAC: Authorizing `identity:get_user(user_id=bb9174aa3ac54d708fcd3d9dc7b9b827)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:41.193904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:41.195533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 9/17] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:56:41 2022] GET /identity/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827 => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:41.201405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:41.215707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:41.245784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b3e43d-f0f1-449c-b3cc-20b566bc0c88 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:41.280971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:41.281762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:41.282055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:41.282340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:41.299782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09aae25-e94c-431a-928c-9dfc86067da1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:41.300137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:41.301648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:56:41.322542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:41.323333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:41.324707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:56:41.324707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:56:41.324707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:56:41.324707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:56:41.324707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 9/18] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:56:41.329982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:41.333524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:41.334325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:41.334614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:41.334898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:41.336080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:41.337571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:41.339177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 10/19] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:41.344717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:41.384712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:41.425073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:41.429178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f750079-33dc-46f8-abb2-3dc301188141 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:41.444888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2fe6a0-55ba-4d0c-b986-9d0738eeffd4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:41.463419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1b858d-ae33-4f82-b674-66a448a30fac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:41.464091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1b858d-ae33-4f82-b674-66a448a30fac None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:41.464380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1b858d-ae33-4f82-b674-66a448a30fac None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:41.464720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1b858d-ae33-4f82-b674-66a448a30fac None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:41.465454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1b858d-ae33-4f82-b674-66a448a30fac None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:41.465763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1b858d-ae33-4f82-b674-66a448a30fac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:41.467175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1b858d-ae33-4f82-b674-66a448a30fac None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:41.478133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3232ea13-3012-408d-8e86-6d10e3736452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:41.478787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3232ea13-3012-408d-8e86-6d10e3736452 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:41.479077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3232ea13-3012-408d-8e86-6d10e3736452 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:41.479359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3232ea13-3012-408d-8e86-6d10e3736452 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:41.480196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3232ea13-3012-408d-8e86-6d10e3736452 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.domain.id=default, user.id=bb9174aa3ac54d708fcd3d9dc7b9b827) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:41.481067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3232ea13-3012-408d-8e86-6d10e3736452 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:41.482524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3232ea13-3012-408d-8e86-6d10e3736452 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:41.484954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 10/20] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:56:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:41.489474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 11/21] 10.222.0.20 () {56 vars in 1322 bytes} [Tue Jul 5 09:56:41 2022] GET /identity/v3/role_assignments?user.id=bb9174aa3ac54d708fcd3d9dc7b9b827&scope.domain.id=default&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 233 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:43.855853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa098ac1-0e64-4a81-b4d7-53b3ddae1621 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:43.856414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa098ac1-0e64-4a81-b4d7-53b3ddae1621 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:43.856521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa098ac1-0e64-4a81-b4d7-53b3ddae1621 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:43.857838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 11/22] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:43.863116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47190268-4126-4953-a143-3eb4cccf7bb0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:43.863510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47190268-4126-4953-a143-3eb4cccf7bb0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:43.863687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47190268-4126-4953-a143-3eb4cccf7bb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:43.889635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-47190268-4126-4953-a143-3eb4cccf7bb0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:43.890873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-47190268-4126-4953-a143-3eb4cccf7bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:43.928025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6e47c-9b22-4f8e-83ed-27d747d38871 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:43.928320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6e47c-9b22-4f8e-83ed-27d747d38871 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:43.928632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6e47c-9b22-4f8e-83ed-27d747d38871 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:43.930175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 12/23] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:43.943580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d90ed8-2f26-49a5-88bf-5b91fabb9cd4 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:43.944174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d90ed8-2f26-49a5-88bf-5b91fabb9cd4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:43.944685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d90ed8-2f26-49a5-88bf-5b91fabb9cd4 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:43.946257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 13/24] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:56:43.947552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0566c7df-a674-4163-bd0d-c4959836c103 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:43.948038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0566c7df-a674-4163-bd0d-c4959836c103 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:43.948539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0566c7df-a674-4163-bd0d-c4959836c103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:43.971525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0566c7df-a674-4163-bd0d-c4959836c103 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:43.973110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0566c7df-a674-4163-bd0d-c4959836c103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:44.008980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 12/25] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:44.010685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adcd60-a31e-4c25-8b51-269125bfe3e4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.011187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adcd60-a31e-4c25-8b51-269125bfe3e4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.012201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adcd60-a31e-4c25-8b51-269125bfe3e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.030070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-85adcd60-a31e-4c25-8b51-269125bfe3e4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:44.031639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-85adcd60-a31e-4c25-8b51-269125bfe3e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:44.105653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 13/26] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:44.107238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85adcd60-a31e-4c25-8b51-269125bfe3e4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:44.122170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:44.122411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 14/27] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:44.145333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0566c7df-a674-4163-bd0d-c4959836c103 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:44.225816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:44.305518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85adcd60-a31e-4c25-8b51-269125bfe3e4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:44.330270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0566c7df-a674-4163-bd0d-c4959836c103 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:44.370356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:44.371123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.371408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.371687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.385523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:44.386202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.386511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.386797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.387561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:44.389169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:44.397342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:44.398835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:56:44.436020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:44.436951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:44.437766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:56:44.437766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:56:44.437766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:56:44.437766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:56:44.437766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 14/28] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:56:44.440733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:44.531912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:44.600188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967285ea-92c6-44e6-a50a-ad4ea3b4af9d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:44.616593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 15/29] 10.222.0.20 () {60 vars in 1131 bytes} [Tue Jul 5 09:56:44 2022] POST /identity/v3/roles => generated 219 bytes in 472 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:56:44.617896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:44.643866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892a187-ea58-43dd-96e7-b9b8ab010bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:44.644669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892a187-ea58-43dd-96e7-b9b8ab010bca None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.644968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892a187-ea58-43dd-96e7-b9b8ab010bca None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.645415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892a187-ea58-43dd-96e7-b9b8ab010bca None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.668291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7892a187-ea58-43dd-96e7-b9b8ab010bca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-7892a187-ea58-43dd-96e7-b9b8ab010bca None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:44.676625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:44.677534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:44.678267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:56:44.679789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 15/30] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:56:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:56:44.688128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3c5ac4-92b7-419b-8240-e08bb1590cbf None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.688490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3c5ac4-92b7-419b-8240-e08bb1590cbf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.688777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3c5ac4-92b7-419b-8240-e08bb1590cbf None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.690274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 16/31] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:44 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:44.691042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3c5ac4-92b7-419b-8240-e08bb1590cbf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:44.807820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13ed06-408a-4919-9943-2ce9844bb761 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:44.825823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:44.853378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce10ca0-dc8e-4438-8876-68a7f8d18d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:44.854211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce10ca0-dc8e-4438-8876-68a7f8d18d58 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.854577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce10ca0-dc8e-4438-8876-68a7f8d18d58 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.854935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce10ca0-dc8e-4438-8876-68a7f8d18d58 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.855804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce10ca0-dc8e-4438-8876-68a7f8d18d58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:44.856202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce10ca0-dc8e-4438-8876-68a7f8d18d58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:44.857862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce10ca0-dc8e-4438-8876-68a7f8d18d58 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:44.858658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3c5ac4-92b7-419b-8240-e08bb1590cbf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:44.872826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 16/32] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:56:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:44.873581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33313cda-bee1-4cf2-b8b2-8f0d05436b63 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.873975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33313cda-bee1-4cf2-b8b2-8f0d05436b63 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.874314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33313cda-bee1-4cf2-b8b2-8f0d05436b63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.880363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:44.893612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-33313cda-bee1-4cf2-b8b2-8f0d05436b63 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:44.895351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-33313cda-bee1-4cf2-b8b2-8f0d05436b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:44.905082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:44.906069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:44.906649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:44.907118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:44.908183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:44.908723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:44.910343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:44.920116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 17/33] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:44.921505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:44.978736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 17/34] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:44.978736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33313cda-bee1-4cf2-b8b2-8f0d05436b63 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:45.076645 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac49a7-f82a-40ab-8c56-600f61f90979 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:45.086060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:45.092624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33313cda-bee1-4cf2-b8b2-8f0d05436b63 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:45.115220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:45.116002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.116298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.116604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.117322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:45.118915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:45.130937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:45.131910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.132227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.132517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] PATH_INFO: `/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.134164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] RBAC: Authorizing `identity:get_user(user_id=bb9174aa3ac54d708fcd3d9dc7b9b827)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:45.135758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:45.138493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 18/35] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:56:44 2022] GET /identity/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827 => generated 247 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:45.143914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:45.248621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7e130-900e-431c-a075-e9bb539a3dc5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:45.276423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 18/36] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:56:44 2022] POST /identity/v3/projects => generated 301 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:56:45.276423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:45.294982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282eac75-80fb-4a8e-90fb-d1027e5eb6ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:45.295829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282eac75-80fb-4a8e-90fb-d1027e5eb6ac None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.296127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282eac75-80fb-4a8e-90fb-d1027e5eb6ac None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.296438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282eac75-80fb-4a8e-90fb-d1027e5eb6ac None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.297303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282eac75-80fb-4a8e-90fb-d1027e5eb6ac None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:45.299069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282eac75-80fb-4a8e-90fb-d1027e5eb6ac None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:45.315912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:45.328640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 19/37] 10.222.0.20 () {60 vars in 1131 bytes} [Tue Jul 5 09:56:45 2022] POST /identity/v3/roles => generated 225 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:56:45.331630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2305af-b6f3-4455-8524-28f3e95d4fb8 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.331980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2305af-b6f3-4455-8524-28f3e95d4fb8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.336085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2305af-b6f3-4455-8524-28f3e95d4fb8 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.338807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 20/38] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:45 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:45.344872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a62795-2148-42ad-93d6-b8fe583ab720 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.345171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a62795-2148-42ad-93d6-b8fe583ab720 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.345464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a62795-2148-42ad-93d6-b8fe583ab720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.386361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-85a62795-2148-42ad-93d6-b8fe583ab720 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:45.388300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-85a62795-2148-42ad-93d6-b8fe583ab720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:45.435967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:45.516142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 21/39] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:45.517879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0600aba-34a6-4f05-8116-33a39fd07a06 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.518179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0600aba-34a6-4f05-8116-33a39fd07a06 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.518468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0600aba-34a6-4f05-8116-33a39fd07a06 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.526538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 22/40] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:45 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:45.532617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6281c34-195b-40bf-bdae-c378924f1aa5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.532921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6281c34-195b-40bf-bdae-c378924f1aa5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.533207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6281c34-195b-40bf-bdae-c378924f1aa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.558267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d6281c34-195b-40bf-bdae-c378924f1aa5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:45.559460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d6281c34-195b-40bf-bdae-c378924f1aa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:45.596969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211f3409-beef-42c6-b3f6-507a8c37cfa5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:45.628434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 23/41] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:45.629344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6281c34-195b-40bf-bdae-c378924f1aa5 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:45.634506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:45.635322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.635617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.635910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.637854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:45.639703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:45.641653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 19/42] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 375 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:45.646858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:45.716834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:45.733793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6281c34-195b-40bf-bdae-c378924f1aa5 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:45.742269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:45.746157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901f5e9e-cae7-48c7-995f-7c45ae6c6cf6 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:45.767150 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:45.767753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.768041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.768324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.769090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:45.770484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:45.780490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:45.781360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.781642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.781918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] PATH_INFO: `/v3/domains/default/users/bb9174aa3ac54d708fcd3d9dc7b9b827/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.783714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 24/43] 10.222.0.20 () {60 vars in 1131 bytes} [Tue Jul 5 09:56:45 2022] POST /identity/v3/roles => generated 223 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:56:45.784007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=bb9174aa3ac54d708fcd3d9dc7b9b827, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:45.784543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:45.785485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:45.808009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 20/44] 10.222.0.20 () {58 vars in 1361 bytes} [Tue Jul 5 09:56:45 2022] PUT /identity/v3/domains/default/users/bb9174aa3ac54d708fcd3d9dc7b9b827/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 09:56:45.914739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e999ac7-4f86-415b-9081-25438f135360 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:45.961292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b426d4-f26b-41cf-960a-6b256271951b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:45.962111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b426d4-f26b-41cf-960a-6b256271951b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:45.962407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b426d4-f26b-41cf-960a-6b256271951b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:45.962690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b426d4-f26b-41cf-960a-6b256271951b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:45.963957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b426d4-f26b-41cf-960a-6b256271951b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:45.966032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b426d4-f26b-41cf-960a-6b256271951b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:45.967706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 25/45] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:45.973285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:46.080151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ba062-5854-4533-a83a-32f5bc93454b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:46.113089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c3a707-dd51-4704-ba47-19f62099143f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:46.113770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c3a707-dd51-4704-ba47-19f62099143f None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:46.114041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c3a707-dd51-4704-ba47-19f62099143f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:46.114320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c3a707-dd51-4704-ba47-19f62099143f None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:46.115039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c3a707-dd51-4704-ba47-19f62099143f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:46.116540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c3a707-dd51-4704-ba47-19f62099143f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:46.141493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 21/46] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:56:45 2022] POST /identity/v3/projects => generated 312 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:56:47.818222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ebf554-8b4c-4a9c-9297-d039f82d416b None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:47.818760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ebf554-8b4c-4a9c-9297-d039f82d416b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:47.818760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ebf554-8b4c-4a9c-9297-d039f82d416b None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:47.820154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 26/47] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:47.825547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38a6f6a-c8e4-4858-bbf4-21d4b4a91031 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:47.825948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38a6f6a-c8e4-4858-bbf4-21d4b4a91031 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:47.826245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38a6f6a-c8e4-4858-bbf4-21d4b4a91031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:47.843103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f38a6f6a-c8e4-4858-bbf4-21d4b4a91031 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:47.844666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f38a6f6a-c8e4-4858-bbf4-21d4b4a91031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:47.847526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:47.912642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 22/48] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:47.936315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ebf554-8b4c-4a9c-9297-d039f82d416b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:47.951463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:48.036968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ebf554-8b4c-4a9c-9297-d039f82d416b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:48.070511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ddb5dd2-4c1c-47f7-a66a-26f14d7446be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:48.071241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb5dd2-4c1c-47f7-a66a-26f14d7446be None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.071530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb5dd2-4c1c-47f7-a66a-26f14d7446be None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.071812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ddb5dd2-4c1c-47f7-a66a-26f14d7446be None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.082806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ddb5dd2-4c1c-47f7-a66a-26f14d7446be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-1ddb5dd2-4c1c-47f7-a66a-26f14d7446be None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:48.092145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:48.093011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:56:48.093826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 27/49] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:56:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:56:48.099103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38a6f6a-c8e4-4858-bbf4-21d4b4a91031 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:48.162250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169cf5b4-f9a8-4ded-bed6-56aae10ee262 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.162540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169cf5b4-f9a8-4ded-bed6-56aae10ee262 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.162824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-169cf5b4-f9a8-4ded-bed6-56aae10ee262 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.164336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 28/50] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:56:48.168893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9307e8d-2ad6-4617-8058-44e878016495 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.169178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9307e8d-2ad6-4617-8058-44e878016495 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.169461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9307e8d-2ad6-4617-8058-44e878016495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.200689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38a6f6a-c8e4-4858-bbf4-21d4b4a91031 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:48.238730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:48.239417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.239702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.239981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.240787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:48.241099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:48.242527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:48.252591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 23/51] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:48.258486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:48.261978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:48.290674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f9307e8d-2ad6-4617-8058-44e878016495 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:48.291906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f9307e8d-2ad6-4617-8058-44e878016495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:48.357704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 29/52] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:48.361557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929904f1-f0f9-4741-95c1-5431a4f38943 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:48.380152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9307e8d-2ad6-4617-8058-44e878016495 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:48.382222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:48.395557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:48.396466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.396755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.397035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] PATH_INFO: `/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.398249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] RBAC: Authorizing `identity:get_user(user_id=bb9174aa3ac54d708fcd3d9dc7b9b827)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:48.399713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:48.401391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 24/53] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:56:48 2022] GET /identity/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827 => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:48.407152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:48.475356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:48.482733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9307e8d-2ad6-4617-8058-44e878016495 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:48.505877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce3b28a-b685-43f8-b4b9-a1970942b4e1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:48.517276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:48.518062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.518347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.518625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.519787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:48.521288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:48.522938 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 30/54] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:48.529033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:48.539617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:48.540418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.540731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.541012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.542180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:48.544120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:48.545898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 25/55] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:48.551256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:48.598815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:48.630321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb95c78-a8a1-4edd-ab1a-d00c88652d77 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:48.653142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40590aee-5ea1-4e52-891d-cf94a031671f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:48.665171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c978bee-9360-4142-82f9-57b7804c241f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:48.665828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c978bee-9360-4142-82f9-57b7804c241f None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.666117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c978bee-9360-4142-82f9-57b7804c241f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.666397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c978bee-9360-4142-82f9-57b7804c241f None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.667104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c978bee-9360-4142-82f9-57b7804c241f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:48.669225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c978bee-9360-4142-82f9-57b7804c241f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:48.688140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f22a9-b24d-4285-846c-e389511c6338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:48.689097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f22a9-b24d-4285-846c-e389511c6338 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:48.689388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f22a9-b24d-4285-846c-e389511c6338 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:48.689676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f22a9-b24d-4285-846c-e389511c6338 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:48.690520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f22a9-b24d-4285-846c-e389511c6338 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.domain.id=default, user.id=bb9174aa3ac54d708fcd3d9dc7b9b827) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:48.690925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f22a9-b24d-4285-846c-e389511c6338 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:48.692285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f22a9-b24d-4285-846c-e389511c6338 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:48.695627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 31/56] 10.222.0.20 () {60 vars in 1140 bytes} [Tue Jul 5 09:56:48 2022] POST /identity/v3/projects => generated 298 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:56:48.698409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 26/57] 10.222.0.20 () {56 vars in 1322 bytes} [Tue Jul 5 09:56:48 2022] GET /identity/v3/role_assignments?user.id=bb9174aa3ac54d708fcd3d9dc7b9b827&scope.domain.id=default&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 530 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:50.709376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b641362-a811-44f8-90d4-1dd90a356de2 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:50.709888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b641362-a811-44f8-90d4-1dd90a356de2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:50.709972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b641362-a811-44f8-90d4-1dd90a356de2 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:50.711309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 32/58] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:50.716865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc929-a5fe-4e97-b102-42f5af53a555 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:50.717283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc929-a5fe-4e97-b102-42f5af53a555 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:50.717432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bc929-a5fe-4e97-b102-42f5af53a555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:50.722456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:50.734861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-884bc929-a5fe-4e97-b102-42f5af53a555 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:50.736052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-884bc929-a5fe-4e97-b102-42f5af53a555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:50.800762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 27/59] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:50.824629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b641362-a811-44f8-90d4-1dd90a356de2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:50.924525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b641362-a811-44f8-90d4-1dd90a356de2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:50.946732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:50.959113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56150bf6-635d-40b5-b723-683c7852b4e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:50.959941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56150bf6-635d-40b5-b723-683c7852b4e7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:50.960180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56150bf6-635d-40b5-b723-683c7852b4e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:50.960476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56150bf6-635d-40b5-b723-683c7852b4e7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:50.961639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56150bf6-635d-40b5-b723-683c7852b4e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:50.963086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56150bf6-635d-40b5-b723-683c7852b4e7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:50.965292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 33/60] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:50.970667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bc929-a5fe-4e97-b102-42f5af53a555 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:51.066003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bc929-a5fe-4e97-b102-42f5af53a555 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:51.094776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:51.099748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e03ce9-6b42-4a5d-9d95-7a4eb552643b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:51.100434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e03ce9-6b42-4a5d-9d95-7a4eb552643b None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:51.100728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e03ce9-6b42-4a5d-9d95-7a4eb552643b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:51.101010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e03ce9-6b42-4a5d-9d95-7a4eb552643b None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:51.101721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e03ce9-6b42-4a5d-9d95-7a4eb552643b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:51.103167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e03ce9-6b42-4a5d-9d95-7a4eb552643b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:51.126906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 28/61] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:56:50 2022] POST /identity/v3/users => generated 275 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:56:54.206881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad03bc2-29cd-4435-b4ec-840356053dab None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.207732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad03bc2-29cd-4435-b4ec-840356053dab None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.208159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad03bc2-29cd-4435-b4ec-840356053dab None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.209727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 34/62] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:54.215459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c99150-5563-4385-8213-01eacd39cd4a None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.216282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c99150-5563-4385-8213-01eacd39cd4a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.216798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c99150-5563-4385-8213-01eacd39cd4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.240534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d4c99150-5563-4385-8213-01eacd39cd4a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:54.241755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d4c99150-5563-4385-8213-01eacd39cd4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:54.300169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55acad-b4fc-419c-b0d5-b891d94a3c35 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.300821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55acad-b4fc-419c-b0d5-b891d94a3c35 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.301289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55acad-b4fc-419c-b0d5-b891d94a3c35 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.302853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 35/63] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:54.307969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14624693-36fe-4e8b-8b0d-d71f9bb0991c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.308537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14624693-36fe-4e8b-8b0d-d71f9bb0991c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.309012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14624693-36fe-4e8b-8b0d-d71f9bb0991c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.334079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-14624693-36fe-4e8b-8b0d-d71f9bb0991c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:54.335297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-14624693-36fe-4e8b-8b0d-d71f9bb0991c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:54.359259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 29/64] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:54.382327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c99150-5563-4385-8213-01eacd39cd4a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:54.427340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 36/65] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:54.463577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14624693-36fe-4e8b-8b0d-d71f9bb0991c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:54.543343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c99150-5563-4385-8213-01eacd39cd4a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:54.567321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:54.589732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f9fd86-cbd6-4f65-8772-fd84ac91ecd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:54.591280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f9fd86-cbd6-4f65-8772-fd84ac91ecd1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.591995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f9fd86-cbd6-4f65-8772-fd84ac91ecd1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.592463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f9fd86-cbd6-4f65-8772-fd84ac91ecd1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.603145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14624693-36fe-4e8b-8b0d-d71f9bb0991c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:54.607812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f9fd86-cbd6-4f65-8772-fd84ac91ecd1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-32f9fd86-cbd6-4f65-8772-fd84ac91ecd1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:54.617438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:54.618304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:56:54.619578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 30/66] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:56:54.620468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d4ef2-fb59-4d82-9759-ed78e140e798 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.620468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d4ef2-fb59-4d82-9759-ed78e140e798 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.620602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7d4ef2-fb59-4d82-9759-ed78e140e798 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.622663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 31/67] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:54.625058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d4ef2-fb59-4d82-9759-ed78e140e798 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:54.635469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:54.655606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e60de3-8dd5-4a01-95f6-d1f4b34e8b80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:54.656648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e60de3-8dd5-4a01-95f6-d1f4b34e8b80 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.657101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e60de3-8dd5-4a01-95f6-d1f4b34e8b80 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.658059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e60de3-8dd5-4a01-95f6-d1f4b34e8b80 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.673988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e60de3-8dd5-4a01-95f6-d1f4b34e8b80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-f7e60de3-8dd5-4a01-95f6-d1f4b34e8b80 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:54.683792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:54.684734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:56:54.685594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 37/68] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:56:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:56:54.686471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0418db2-298a-46da-b2da-398090556b5c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.686720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0418db2-298a-46da-b2da-398090556b5c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.687159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0418db2-298a-46da-b2da-398090556b5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.704212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e0418db2-298a-46da-b2da-398090556b5c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:54.705618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e0418db2-298a-46da-b2da-398090556b5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:54.746026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7d4ef2-fb59-4d82-9759-ed78e140e798 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:54.782285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cdc5fb-29a0-4f81-bb31-736a585d95b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:54.782972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cdc5fb-29a0-4f81-bb31-736a585d95b2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.783263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cdc5fb-29a0-4f81-bb31-736a585d95b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.783544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cdc5fb-29a0-4f81-bb31-736a585d95b2 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.784305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cdc5fb-29a0-4f81-bb31-736a585d95b2 None admin] RBAC: Adding query filter params (name=member) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:54.784656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cdc5fb-29a0-4f81-bb31-736a585d95b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:54.793883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 38/69] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:54.794051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cdc5fb-29a0-4f81-bb31-736a585d95b2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:54.794137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f408c5e-8045-453b-ad51-62e6e071844f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.794137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f408c5e-8045-453b-ad51-62e6e071844f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.794137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f408c5e-8045-453b-ad51-62e6e071844f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.794137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 39/70] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:54.794137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f408c5e-8045-453b-ad51-62e6e071844f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:54.802618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 32/71] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:56:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:54.803949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168e6cdf-ff65-4755-9717-e4a582eaf6be None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.804247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168e6cdf-ff65-4755-9717-e4a582eaf6be None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.804680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168e6cdf-ff65-4755-9717-e4a582eaf6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.822213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-168e6cdf-ff65-4755-9717-e4a582eaf6be None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:54.823526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-168e6cdf-ff65-4755-9717-e4a582eaf6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:54.861008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:54.894546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f408c5e-8045-453b-ad51-62e6e071844f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:54.897205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 33/72] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:54.898053 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168e6cdf-ff65-4755-9717-e4a582eaf6be None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:54.915579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:54.948725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:54.949425 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:54.949719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:54.950007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:54.950763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:54.951075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:54.952502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:54.963833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 40/73] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:54.965188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.006459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168e6cdf-ff65-4755-9717-e4a582eaf6be None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.041261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.042142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.042428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.042737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.055797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.057304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.059001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 34/74] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:56:54 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:55.059768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.068860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc4b3a-bcb1-42c6-bb89-22456f637dde None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.087366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.103863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009e171a-6fc0-4bd3-bc9f-f7798cae3c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.104701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e171a-6fc0-4bd3-bc9f-f7798cae3c1e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.104985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e171a-6fc0-4bd3-bc9f-f7798cae3c1e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.105287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009e171a-6fc0-4bd3-bc9f-f7798cae3c1e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.115184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009e171a-6fc0-4bd3-bc9f-f7798cae3c1e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-009e171a-6fc0-4bd3-bc9f-f7798cae3c1e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:55.123233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:55.124228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:56:55.125233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 41/75] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:56:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:56:55.125923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad6a7ec-f5c8-465d-aef3-16046fe0bd41 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.126212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad6a7ec-f5c8-465d-aef3-16046fe0bd41 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.126491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad6a7ec-f5c8-465d-aef3-16046fe0bd41 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.127906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 42/76] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:55.128489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad6a7ec-f5c8-465d-aef3-16046fe0bd41 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.160493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.164807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9e2561-6664-4146-901f-4b95ab831e79 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.197583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.198386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.198671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.199501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.209526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:55.218155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:55.224599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 35/77] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:56:55.225372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.232036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad6a7ec-f5c8-465d-aef3-16046fe0bd41 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.266733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.267538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.267827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.268108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] PATH_INFO: `/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.269341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] RBAC: Authorizing `identity:get_user(user_id=bb9174aa3ac54d708fcd3d9dc7b9b827)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.270806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.272449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 43/78] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:55.273218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.315099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.329079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a599290-37ff-4e6d-9fce-3d6d165115a5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.362186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.362849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.363140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.363516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.367168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.370188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e0bb0f-ebe8-4470-8507-6dcc2cf64327 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.404838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0d4662-3bba-4b94-bd06-48a352a040fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.405500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d4662-3bba-4b94-bd06-48a352a040fc None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.405794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d4662-3bba-4b94-bd06-48a352a040fc None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.406073 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0d4662-3bba-4b94-bd06-48a352a040fc None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.406827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0d4662-3bba-4b94-bd06-48a352a040fc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:55.407136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0d4662-3bba-4b94-bd06-48a352a040fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.408535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0d4662-3bba-4b94-bd06-48a352a040fc None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.416487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 44/79] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:55.417788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e37bc5-65b7-4a27-9ae3-97a87a0f85fe None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.418260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e37bc5-65b7-4a27-9ae3-97a87a0f85fe None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.418543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e37bc5-65b7-4a27-9ae3-97a87a0f85fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.434423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-17e37bc5-65b7-4a27-9ae3-97a87a0f85fe None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:55.435464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.435617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-17e37bc5-65b7-4a27-9ae3-97a87a0f85fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:55.436999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.501310 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 45/80] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:55.502341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e37bc5-65b7-4a27-9ae3-97a87a0f85fe None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.524665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.543290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 36/81] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:56:55.543290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.606839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.615521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e37bc5-65b7-4a27-9ae3-97a87a0f85fe None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.638184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01eb371-319e-4d9e-b6aa-ad3e6bf51e64 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.649775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.650445 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.650737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.651021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.651774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:55.652655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.654057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.662026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 46/82] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:55.662822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.677489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.678147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.678444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.678725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.733786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.746597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.748087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.759883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23dda50-cd1c-4b86-97c7-12b83cb1bc1a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.794345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.795152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.795441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.795729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.796985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.798460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.800109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 47/83] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:55.800906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.825861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.853171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 37/84] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:56:55.854008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.903589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5b1451-a583-43a1-bef3-bb15eebd863b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.935402 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.936198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.936521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.936802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.938013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.939477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:55.941124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 48/85] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:55.941245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:55.945767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:55.957577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd7def2-3025-43e9-ba52-4008eb31a75e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:55.992642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:55.993512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:55.993806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:55.994082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:55.995276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:55.997381 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.003763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 38/86] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:56.003763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.040652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:56.050865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1594c4f-0d2f-43fb-8153-c92ecf4b8b92 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.083270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.084223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.084551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.084840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.086040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.087492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.089164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 49/87] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:56.089935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.102234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc466ff9-d98c-4944-8885-9845706eb1b4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.136063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.136793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.137080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.137363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.161846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:56.188222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3973b455-6234-4249-ade5-78ae5efc2768 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.204405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.205885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.223564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.224444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.224734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.225013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.226191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.227628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.229352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 50/88] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:56:56 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:56.230107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.257000 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:56.309929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 39/89] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:56:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:56:56.311345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.330080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0fda63-0118-4123-aea8-e9e9bc49251e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.363192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.363859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.364146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.364445 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.365160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.366461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:56.366599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.391685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 51/90] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:56:56 2022] POST /identity/v3/users => generated 289 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:56:56.392627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.428239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cb58c3-ed92-45e0-ad5b-65b99b1effcc None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.468991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.469678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.469959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.470397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.471272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=02951879c7ac4670a6f8f5177df11cb3) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:56.471594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.473141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.484425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 40/91] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:56:56 2022] GET /identity/v3/role_assignments?user.id=02951879c7ac4670a6f8f5177df11cb3&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:56.484425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.499894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:56.526741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a087d13-3e18-4b41-aef1-35e2cff2dc8b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.576529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.577430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.577862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.578317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.630958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356f2ccf-46a0-40e6-b07f-6093fe6c6e8f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.657998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:56.675889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.676693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.677159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.677575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.678563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=bb9174aa3ac54d708fcd3d9dc7b9b827) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:56.679072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.680743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.683133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.685262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.688425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 41/92] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:56:56 2022] GET /identity/v3/role_assignments?user.id=bb9174aa3ac54d708fcd3d9dc7b9b827&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:56.694087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.789015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:56.803230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 52/93] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:56:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:56:56.809177 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:56.933843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3789043b-7775-44f2-968a-4ab700e0eca4 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.935451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04610309-9d07-4f81-b357-550370fa0dc3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:56.980889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:56.981919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:56.982364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:56.982785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:56.986650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:56.988331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:56.990729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 42/94] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:56:56 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:56.997291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:57.055753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:57.056950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:57.057453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:57.057886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] PATH_INFO: `/v3/projects/1c57194cea024b6e8fdc9a2d5910c313` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:57.061155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] RBAC: Authorizing `identity:get_project(project_id=1c57194cea024b6e8fdc9a2d5910c313)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:57.062781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:57.073148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 53/95] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:56:56 2022] GET /identity/v3/projects/1c57194cea024b6e8fdc9a2d5910c313 => generated 312 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:57.081389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:57.087476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:57.178041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:57.185704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ea031f-281b-4ffd-b6f6-07ec21d5e381 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:57.274549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c7ab39-c66e-4201-a52e-a1ef9ac94321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:57.275224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c7ab39-c66e-4201-a52e-a1ef9ac94321 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:57.275514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c7ab39-c66e-4201-a52e-a1ef9ac94321 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:57.275797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c7ab39-c66e-4201-a52e-a1ef9ac94321 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:57.276678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c7ab39-c66e-4201-a52e-a1ef9ac94321 None admin] RBAC: Adding query filter params (role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=02951879c7ac4670a6f8f5177df11cb3) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:57.277048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c7ab39-c66e-4201-a52e-a1ef9ac94321 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:57.279020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c7ab39-c66e-4201-a52e-a1ef9ac94321 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:57.283637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7202bee-b8b1-4932-a460-ed026f2d34ba None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:57.298209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 43/96] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:56:56 2022] GET /identity/v3/role_assignments?user.id=02951879c7ac4670a6f8f5177df11cb3&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 259 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:56:57.370321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c07b24-ebdd-4ef1-ae1c-2bebc5fe9b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:57.371586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c07b24-ebdd-4ef1-ae1c-2bebc5fe9b1e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:57.371877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c07b24-ebdd-4ef1-ae1c-2bebc5fe9b1e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:57.372160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c07b24-ebdd-4ef1-ae1c-2bebc5fe9b1e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:57.373056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c07b24-ebdd-4ef1-ae1c-2bebc5fe9b1e None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=1c57194cea024b6e8fdc9a2d5910c313, user.id=02951879c7ac4670a6f8f5177df11cb3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:56:57.373374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c07b24-ebdd-4ef1-ae1c-2bebc5fe9b1e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:57.374841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c07b24-ebdd-4ef1-ae1c-2bebc5fe9b1e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:57.387289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 54/97] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:56:57 2022] GET /identity/v3/role_assignments?user.id=02951879c7ac4670a6f8f5177df11cb3&scope.project.id=1c57194cea024b6e8fdc9a2d5910c313&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 259 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:59.343702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83397ab-a730-4297-aeb0-5a1a2af156ae None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.344215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83397ab-a730-4297-aeb0-5a1a2af156ae None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.344289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83397ab-a730-4297-aeb0-5a1a2af156ae None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.345700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 44/98] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:59.351022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735e69a-39d5-439b-a03d-8b3018291cca None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.351415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735e69a-39d5-439b-a03d-8b3018291cca None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.351581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735e69a-39d5-439b-a03d-8b3018291cca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.375317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5735e69a-39d5-439b-a03d-8b3018291cca None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:59.376537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5735e69a-39d5-439b-a03d-8b3018291cca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:59.379425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1615b8-2977-4ba0-94ea-496fec13b251 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.379943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1615b8-2977-4ba0-94ea-496fec13b251 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.380458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1615b8-2977-4ba0-94ea-496fec13b251 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.382288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 45/99] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:59.389306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fd86a0-d00b-49a8-92f5-067fa3560bbb None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.390120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fd86a0-d00b-49a8-92f5-067fa3560bbb None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.391036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fd86a0-d00b-49a8-92f5-067fa3560bbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.413582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-38fd86a0-d00b-49a8-92f5-067fa3560bbb None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:59.415516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-38fd86a0-d00b-49a8-92f5-067fa3560bbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:59.466861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 55/100] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:56:59.468534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06185569-ec19-4c4b-99dc-22648aade7a4 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.468830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06185569-ec19-4c4b-99dc-22648aade7a4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.469178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06185569-ec19-4c4b-99dc-22648aade7a4 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.470586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 56/101] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:56:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:56:59.475772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd526d2-4538-4ba3-93ac-87424427cd98 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.476245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd526d2-4538-4ba3-93ac-87424427cd98 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.476721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd526d2-4538-4ba3-93ac-87424427cd98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.497170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-6bd526d2-4538-4ba3-93ac-87424427cd98 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:56:59.499143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6bd526d2-4538-4ba3-93ac-87424427cd98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:56:59.503364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 46/102] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:59.504312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fd86a0-d00b-49a8-92f5-067fa3560bbb None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:59.557537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:59.576646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:59.596566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 57/103] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:56:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:56:59.597770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd526d2-4538-4ba3-93ac-87424427cd98 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:59.649323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fd86a0-d00b-49a8-92f5-067fa3560bbb None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:59.698541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:59.699903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.700194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.700504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.715410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:59.724533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:59.725434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:56:59.726173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:56:59.728590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 47/104] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:56:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:56:59.729803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:59.731900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd526d2-4538-4ba3-93ac-87424427cd98 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:59.787023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:59.788198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.789514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.790583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.810295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:56:59.825334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:56:59.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:56:59.826914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:56:59.833158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 58/105] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:56:59.834195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:59.880636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:59.902001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddf9d58-e3bb-4373-8441-4b42cf5c33ed None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:56:59.913961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:56:59.953711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:56:59.954858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:56:59.955343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:56:59.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:56:59.957304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:56:59.958964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:56:59.960856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 48/106] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:56:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:56:59.961818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:56:59.980887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc340ae-ff34-493c-ab6f-f86083d19053 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.016246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4442a625-1ced-49bb-970b-dfea0f48fccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.016948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442a625-1ced-49bb-970b-dfea0f48fccf None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.017237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442a625-1ced-49bb-970b-dfea0f48fccf None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.017521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4442a625-1ced-49bb-970b-dfea0f48fccf None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.018329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4442a625-1ced-49bb-970b-dfea0f48fccf None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:00.018641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4442a625-1ced-49bb-970b-dfea0f48fccf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.020086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4442a625-1ced-49bb-970b-dfea0f48fccf None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.029738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 59/107] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:56:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:00.031473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b028fca-ae13-4531-8bd2-3aff94972326 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.031932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b028fca-ae13-4531-8bd2-3aff94972326 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.032380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b028fca-ae13-4531-8bd2-3aff94972326 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.033887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 60/108] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:00.034744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b028fca-ae13-4531-8bd2-3aff94972326 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.070744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b965da-cbea-4b33-a36e-029abdddf37a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.108405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.109098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.109386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.109667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.110429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] RBAC: Adding query filter params (name=member) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:00.110743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.112149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.120740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 49/109] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:56:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:00.125975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.137578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.146359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b028fca-ae13-4531-8bd2-3aff94972326 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.162610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.185525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd90d0-3c91-40bf-90f0-cffa782343ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.186189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd90d0-3c91-40bf-90f0-cffa782343ce None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.186479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd90d0-3c91-40bf-90f0-cffa782343ce None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.186759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd90d0-3c91-40bf-90f0-cffa782343ce None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.187457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd90d0-3c91-40bf-90f0-cffa782343ce None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.188958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd90d0-3c91-40bf-90f0-cffa782343ce None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.216592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 61/110] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:57:00 2022] POST /identity/v3/projects => generated 302 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:00.217855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3480fa0-e64b-4c3e-bf96-d26673721f44 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.218157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3480fa0-e64b-4c3e-bf96-d26673721f44 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.218666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3480fa0-e64b-4c3e-bf96-d26673721f44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.241836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b3480fa0-e64b-4c3e-bf96-d26673721f44 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:00.243241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b3480fa0-e64b-4c3e-bf96-d26673721f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:00.246716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dee45f3-5126-48e1-a6ed-fc342bb1dea2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.290271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bda7d7-d8bc-4551-be4d-f260719247a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.291433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bda7d7-d8bc-4551-be4d-f260719247a2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.291812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bda7d7-d8bc-4551-be4d-f260719247a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.292224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bda7d7-d8bc-4551-be4d-f260719247a2 None admin] PATH_INFO: `/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.293802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bda7d7-d8bc-4551-be4d-f260719247a2 None admin] RBAC: Authorizing `identity:get_user(user_id=bb9174aa3ac54d708fcd3d9dc7b9b827)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.295583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bda7d7-d8bc-4551-be4d-f260719247a2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.298064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 50/111] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827 => generated 247 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:00.299320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c118123-928e-4681-91aa-79e11d8f8b05 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.299856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c118123-928e-4681-91aa-79e11d8f8b05 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.300186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c118123-928e-4681-91aa-79e11d8f8b05 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.301645 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 51/112] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:00.302532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c118123-928e-4681-91aa-79e11d8f8b05 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.326513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 62/113] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:00.328141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca8bc1-000d-4baf-ab09-2c4477bdb1cd None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.328606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca8bc1-000d-4baf-ab09-2c4477bdb1cd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.329033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bca8bc1-000d-4baf-ab09-2c4477bdb1cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.346401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0bca8bc1-000d-4baf-ab09-2c4477bdb1cd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:00.347635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0bca8bc1-000d-4baf-ab09-2c4477bdb1cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:00.428879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 63/114] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:00.429773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca8bc1-000d-4baf-ab09-2c4477bdb1cd None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.441350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.452035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c118123-928e-4681-91aa-79e11d8f8b05 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.456485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.494740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.495626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.496576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.496860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.498417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.499908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.502347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 52/115] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:00.503044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.556165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bca8bc1-000d-4baf-ab09-2c4477bdb1cd None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.592513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.593688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.594160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.594576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.595916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.597662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.599536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 64/116] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:00.600538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.608847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9da77f0-8050-4262-9fc2-e541edf5545a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.650682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.651713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.652178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.652447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.662679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:00.670405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:00.671264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:00.672106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 53/117] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:00.673429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.693641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.709471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.721047 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88cca56-b2e0-419e-8096-05abb3de7558 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.757841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.758677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.758968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.759251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.770711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:00.779019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:00.779866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:00.780639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 65/118] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:00.781251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.794764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669625b7-8e1b-4060-b8fb-712db9e85381 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.831400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8848ecd-6253-468f-b310-7ceb8ee445e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.832285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8848ecd-6253-468f-b310-7ceb8ee445e4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.832603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8848ecd-6253-468f-b310-7ceb8ee445e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.832893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8848ecd-6253-468f-b310-7ceb8ee445e4 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.834057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8848ecd-6253-468f-b310-7ceb8ee445e4 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.835521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8848ecd-6253-468f-b310-7ceb8ee445e4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.837428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 54/119] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:00.838587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83448f3-8b83-4cb1-9cde-9dffbd3272ac None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.838877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83448f3-8b83-4cb1-9cde-9dffbd3272ac None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.839164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83448f3-8b83-4cb1-9cde-9dffbd3272ac None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.840568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 55/120] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:00.841211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83448f3-8b83-4cb1-9cde-9dffbd3272ac None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.891179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a040edc-83ec-4dfa-8b07-5c69aa7f9a96 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:00.929344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:00.930247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:00.930557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:00.930829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8/users/bb9174aa3ac54d708fcd3d9dc7b9b827/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:00.932814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] RBAC: Authorizing `identity:create_grant(project_id=68b3ea181434496b91ea5053896483e8, user_id=bb9174aa3ac54d708fcd3d9dc7b9b827, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:00.934867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:00.939489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.951905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:00.960475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 66/121] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:00 2022] PUT /identity/v3/projects/68b3ea181434496b91ea5053896483e8/users/bb9174aa3ac54d708fcd3d9dc7b9b827/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:00.960475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:00.961810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83448f3-8b83-4cb1-9cde-9dffbd3272ac None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.015431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.016257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.016599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.016884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.017825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] RBAC: Adding query filter params (name=reader) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:01.018472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.019919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.033828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 56/122] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:01.034773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.075250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2442c546-eaaa-4d2a-9d42-912984bba5b5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.113939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8790396e-e297-4595-b096-aaf0f08cb903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.114994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8790396e-e297-4595-b096-aaf0f08cb903 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.116128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8790396e-e297-4595-b096-aaf0f08cb903 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.116468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8790396e-e297-4595-b096-aaf0f08cb903 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.117593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8790396e-e297-4595-b096-aaf0f08cb903 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:01.117913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8790396e-e297-4595-b096-aaf0f08cb903 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.120156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8790396e-e297-4595-b096-aaf0f08cb903 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.132924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 67/123] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:01.134236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d998a9e-e393-4a7c-8e83-ecd840908578 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.134539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d998a9e-e393-4a7c-8e83-ecd840908578 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.134828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d998a9e-e393-4a7c-8e83-ecd840908578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.153291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5d998a9e-e393-4a7c-8e83-ecd840908578 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:01.154514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5d998a9e-e393-4a7c-8e83-ecd840908578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:01.166676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04a5c20-1213-4d8d-ad0e-000ed6ddd8dc None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.209646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.210577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.210917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.211209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8/users/02951879c7ac4670a6f8f5177df11cb3/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.213368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] RBAC: Authorizing `identity:create_grant(project_id=68b3ea181434496b91ea5053896483e8, user_id=02951879c7ac4670a6f8f5177df11cb3, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.214924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.224366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:01.232748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 57/124] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:01 2022] PUT /identity/v3/projects/68b3ea181434496b91ea5053896483e8/users/02951879c7ac4670a6f8f5177df11cb3/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:01.233490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.240470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 68/125] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:01.241365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d998a9e-e393-4a7c-8e83-ecd840908578 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.280329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:01.356556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d998a9e-e393-4a7c-8e83-ecd840908578 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.375616 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48bf015-4e58-4c34-afea-59db3e044a7b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.395833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.396762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.397047 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.397329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.398723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.400237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.402737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 69/126] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:01 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:01.403536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.414187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.415096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.415387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.415729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] PATH_INFO: `/v3/users/6ccea72aef8d4923b23c0da0e9bf8a5f` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.431794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] RBAC: Authorizing `identity:get_user(user_id=6ccea72aef8d4923b23c0da0e9bf8a5f)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.433639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.435515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 58/127] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:01 2022] GET /identity/v3/users/6ccea72aef8d4923b23c0da0e9bf8a5f => generated 289 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:01.436314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.466378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:01.517886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c79025-4912-41ee-84cc-eba91e67009a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.530552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:01.555034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b33071-acc4-49f4-9ffb-5e974ee27afb None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.559227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.560283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.560808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.561276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.573025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:01.583281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:01.584126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:01.584904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:01.587349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 70/128] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:01.588344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.599520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.600435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.600787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.601071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] PATH_INFO: `/v3/projects/1c57194cea024b6e8fdc9a2d5910c313` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.602350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] RBAC: Authorizing `identity:get_project(project_id=1c57194cea024b6e8fdc9a2d5910c313)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.603901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.606097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 59/129] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:01 2022] GET /identity/v3/projects/1c57194cea024b6e8fdc9a2d5910c313 => generated 312 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:01.606890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.722058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af46fcf5-d200-44e5-88d3-1801526781c9 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.723295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5806d58-4928-42fb-b659-9d8e75f47f51 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:01.743873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:01.762199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.762975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.763274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.763567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.765245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:01.765577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.767035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.769812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:01.770693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:01.771171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:01.771655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:01.788315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 60/130] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:01.789350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:01.817602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:01.903079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:01.905866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:01.986091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995dbac2-df87-407e-88e4-b45848e54c9e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:02.025908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:02.026806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:02.027719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:02.028000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] PATH_INFO: `/v3/projects/1c57194cea024b6e8fdc9a2d5910c313/users/02951879c7ac4670a6f8f5177df11cb3/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:02.030127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] RBAC: Authorizing `identity:create_grant(project_id=1c57194cea024b6e8fdc9a2d5910c313, user_id=02951879c7ac4670a6f8f5177df11cb3, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:02.031783 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:02.049765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 61/131] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:01 2022] PUT /identity/v3/projects/1c57194cea024b6e8fdc9a2d5910c313/users/02951879c7ac4670a6f8f5177df11cb3/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:02.051626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:02.109881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 71/132] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 522 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:02.115483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:02.153688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:02.190273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:02.263577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a83e26f-d1fe-4bc7-a0de-a34894fb5a4e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:02.313163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:02.314062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:02.314349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:02.314629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:02.316906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:02.318449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:02.322844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 62/133] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:02 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:02.324996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c09850f-f940-47ce-811d-759fe30e9ad7 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:02.325876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:02.366544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:02.367787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:02.368301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:02.368804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:02.370248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:02.372987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:02.376129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 72/134] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:02 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:02.381480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:02.522047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10663749-36be-476e-b442-365e74bccfd3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:02.584690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:02.596384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:02.597289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:02.597577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:02.597860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:02.599780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:02.601288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:02.603031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 63/135] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:02 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:02.615399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba701f9-e850-45d6-bc23-753ddd5c6c0d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:02.626641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:02.636009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:02.692230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a4feb6-7121-4a9e-bc29-b80cdc5c43d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:02.693355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4feb6-7121-4a9e-bc29-b80cdc5c43d3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:02.693652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4feb6-7121-4a9e-bc29-b80cdc5c43d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:02.693934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4feb6-7121-4a9e-bc29-b80cdc5c43d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:02.694792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a4feb6-7121-4a9e-bc29-b80cdc5c43d3 None admin] RBAC: Adding query filter params (role.id=bf3b74a9a9db444a8288ca37ccf736b7, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=6ccea72aef8d4923b23c0da0e9bf8a5f) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:02.695269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a4feb6-7121-4a9e-bc29-b80cdc5c43d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:02.696875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a4feb6-7121-4a9e-bc29-b80cdc5c43d3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:02.710816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 73/136] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:02 2022] GET /identity/v3/role_assignments?user.id=6ccea72aef8d4923b23c0da0e9bf8a5f&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=bf3b74a9a9db444a8288ca37ccf736b7 => generated 259 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:02.836022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad100f4-daac-46da-942e-96fb1309b190 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:02.883594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8681e1da-bdd2-4c1d-a1d0-b23d83ac8201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:02.884953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e1da-bdd2-4c1d-a1d0-b23d83ac8201 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:02.885495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e1da-bdd2-4c1d-a1d0-b23d83ac8201 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:02.886019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e1da-bdd2-4c1d-a1d0-b23d83ac8201 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8/users/02951879c7ac4670a6f8f5177df11cb3/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:02.889196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8681e1da-bdd2-4c1d-a1d0-b23d83ac8201 None admin] RBAC: Authorizing `identity:create_grant(project_id=68b3ea181434496b91ea5053896483e8, user_id=02951879c7ac4670a6f8f5177df11cb3, role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:02.891183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8681e1da-bdd2-4c1d-a1d0-b23d83ac8201 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:02.908851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 64/137] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:02 2022] PUT /identity/v3/projects/68b3ea181434496b91ea5053896483e8/users/02951879c7ac4670a6f8f5177df11cb3/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 09:57:03.108051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cd2312-d742-48be-8d7c-d2ffb46c02b1 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:03.108582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cd2312-d742-48be-8d7c-d2ffb46c02b1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:03.109013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cd2312-d742-48be-8d7c-d2ffb46c02b1 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:03.110877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 74/138] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:03.116003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f8be42-ca4e-47b4-9ef0-97195afce04f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:03.116491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f8be42-ca4e-47b4-9ef0-97195afce04f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:03.116919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f8be42-ca4e-47b4-9ef0-97195afce04f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:03.138500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-37f8be42-ca4e-47b4-9ef0-97195afce04f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:03.140167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-37f8be42-ca4e-47b4-9ef0-97195afce04f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:03.207302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:03.216674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 65/139] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:03.241274 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cd2312-d742-48be-8d7c-d2ffb46c02b1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:03.384467 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cd2312-d742-48be-8d7c-d2ffb46c02b1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:03.423081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07168405-481c-46ce-8123-623347cedf97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:03.423882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07168405-481c-46ce-8123-623347cedf97 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:03.424165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07168405-481c-46ce-8123-623347cedf97 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:03.424465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07168405-481c-46ce-8123-623347cedf97 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:03.425638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07168405-481c-46ce-8123-623347cedf97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:03.427108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07168405-481c-46ce-8123-623347cedf97 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:03.429175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 75/140] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:03.436926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f8be42-ca4e-47b4-9ef0-97195afce04f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:03.568441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f8be42-ca4e-47b4-9ef0-97195afce04f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:03.611480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7512db98-05f8-4c49-b68c-8607ddf2fcf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:03.612289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7512db98-05f8-4c49-b68c-8607ddf2fcf9 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:03.612477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7512db98-05f8-4c49-b68c-8607ddf2fcf9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:03.612770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7512db98-05f8-4c49-b68c-8607ddf2fcf9 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:03.613500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7512db98-05f8-4c49-b68c-8607ddf2fcf9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:03.614983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7512db98-05f8-4c49-b68c-8607ddf2fcf9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:03.643854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 66/141] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:03 2022] POST /identity/v3/users => generated 283 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:03.818508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b0ada5-3adf-4dac-a4a3-955e9086e8e0 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:03.819034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b0ada5-3adf-4dac-a4a3-955e9086e8e0 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:03.819123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b0ada5-3adf-4dac-a4a3-955e9086e8e0 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:03.820778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 76/142] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:03.826144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a5dc6-690b-4bd0-a228-801a69784f41 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:03.826453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a5dc6-690b-4bd0-a228-801a69784f41 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:03.826738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635a5dc6-690b-4bd0-a228-801a69784f41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:03.845451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-635a5dc6-690b-4bd0-a228-801a69784f41 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:03.846655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-635a5dc6-690b-4bd0-a228-801a69784f41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:03.915396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:03.961121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 67/143] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:03.994698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b0ada5-3adf-4dac-a4a3-955e9086e8e0 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:04.022941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:04.128213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b0ada5-3adf-4dac-a4a3-955e9086e8e0 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:04.169408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcaef00d-cefc-4f49-a848-11740db89c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:04.170137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaef00d-cefc-4f49-a848-11740db89c9c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:04.170424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaef00d-cefc-4f49-a848-11740db89c9c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:04.170704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcaef00d-cefc-4f49-a848-11740db89c9c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:04.186111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcaef00d-cefc-4f49-a848-11740db89c9c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-dcaef00d-cefc-4f49-a848-11740db89c9c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:04.193906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:04.194783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:04.195690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 77/144] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:04.201670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635a5dc6-690b-4bd0-a228-801a69784f41 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:04.346738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf9853c-0c2f-4297-8e2a-7d9829775d86 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:04.347158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf9853c-0c2f-4297-8e2a-7d9829775d86 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:04.347338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf9853c-0c2f-4297-8e2a-7d9829775d86 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:04.349069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 78/145] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:04.354886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90a19c8-19ad-402d-98b3-fe62cfb430b6 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:04.355210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90a19c8-19ad-402d-98b3-fe62cfb430b6 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:04.355370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635a5dc6-690b-4bd0-a228-801a69784f41 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:04.355529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90a19c8-19ad-402d-98b3-fe62cfb430b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:04.389612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c90a19c8-19ad-402d-98b3-fe62cfb430b6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:04.391115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c90a19c8-19ad-402d-98b3-fe62cfb430b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:04.398622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:04.399311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:04.399600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:04.399882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:04.400693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:04.401005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:04.403624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:04.414047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 68/146] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:04.420569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:04.483047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:04.512178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 79/147] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:04.566495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922c7a52-aca9-459e-87e3-b495fe30f3cd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:04.594832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90a19c8-19ad-402d-98b3-fe62cfb430b6 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:04.603611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:04.632512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:04.633426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:04.633718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:04.634006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] PATH_INFO: `/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:04.635326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] RBAC: Authorizing `identity:get_user(user_id=bb9174aa3ac54d708fcd3d9dc7b9b827)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:04.637342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:04.639231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 69/148] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:04 2022] GET /identity/v3/users/bb9174aa3ac54d708fcd3d9dc7b9b827 => generated 247 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:04.645543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:04.714278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90a19c8-19ad-402d-98b3-fe62cfb430b6 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:04.752796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:04.753605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:04.753896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:04.754180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:04.778075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:04.786380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:04.787243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:04.788008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 80/149] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:04.797557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:04.834126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bccade-f2e9-4dbd-8cfc-d89dd9e5b076 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:04.864777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:04.884579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1972a11-9c20-43a1-85bd-74d757f25629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:04.885510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1972a11-9c20-43a1-85bd-74d757f25629 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:04.885962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1972a11-9c20-43a1-85bd-74d757f25629 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:04.886926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1972a11-9c20-43a1-85bd-74d757f25629 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:04.962517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:05.008315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64eec521-1a1a-40b9-87ee-18c4c00f5c55 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:05.013697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1972a11-9c20-43a1-85bd-74d757f25629 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:05.017696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1972a11-9c20-43a1-85bd-74d757f25629 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:05.051450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ffc953-8a9a-44c6-aad9-813185366126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:05.052178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ffc953-8a9a-44c6-aad9-813185366126 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.054760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ffc953-8a9a-44c6-aad9-813185366126 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.055078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ffc953-8a9a-44c6-aad9-813185366126 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.056010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ffc953-8a9a-44c6-aad9-813185366126 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:05.056362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ffc953-8a9a-44c6-aad9-813185366126 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:05.057871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ffc953-8a9a-44c6-aad9-813185366126 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:05.067623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 81/150] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:05.068987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950c20d-be2e-4a41-930a-835f893e58af None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.069280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950c20d-be2e-4a41-930a-835f893e58af None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.069564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950c20d-be2e-4a41-930a-835f893e58af None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.074709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 82/151] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:05.074709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950c20d-be2e-4a41-930a-835f893e58af None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:05.162808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 70/152] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:57:05.164176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f85fa-9dea-40b1-963a-541d1c60f4da None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.164602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f85fa-9dea-40b1-963a-541d1c60f4da None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.165037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f85fa-9dea-40b1-963a-541d1c60f4da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.195213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b62f85fa-9dea-40b1-963a-541d1c60f4da None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:05.196672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b62f85fa-9dea-40b1-963a-541d1c60f4da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:05.210419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:05.228580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950c20d-be2e-4a41-930a-835f893e58af None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:05.277948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:05.288797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:05.290241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.290883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.291362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.293731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:05.295412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:05.297776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 83/153] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:05 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:05.303451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:05.332329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 71/154] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:05.333554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f85fa-9dea-40b1-963a-541d1c60f4da None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:05.468222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f14a6-e009-47c0-a916-0beb704fc53f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:05.481478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f85fa-9dea-40b1-963a-541d1c60f4da None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:05.510473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:05.511892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.512180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.513913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.515580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:05.517250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:05.519153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 84/155] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:05 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:05.520068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:05.537281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:05.538234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.538737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.539303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.560580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:05.584536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:05.665723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:05.667243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:05.681663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0f388-c45a-4c92-8722-38509e412cfd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:05.736938 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:05.738086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.738573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.739068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.754215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:05.767456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:05.768568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:05.769449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:05.778886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 85/156] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:05.779998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:05.833197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 72/157] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:05.833197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:05.920417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:05.930269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dfc620-ce4b-4ff8-a9c4-336cf5bf48fe None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:05.970124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:05.992416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2f363-ed52-4918-82df-2868dc4c5611 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:05.993726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:05.994647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:05.995142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:05.996155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:05.997252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=bb9174aa3ac54d708fcd3d9dc7b9b827) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:05.997785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:05.999478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:06.010477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 86/158] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:05 2022] GET /identity/v3/role_assignments?user.id=bb9174aa3ac54d708fcd3d9dc7b9b827&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:06.012341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:06.081199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:06.082043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.082169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.082433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.083267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] RBAC: Adding query filter params (name=member) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:06.083514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:06.084952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:06.094098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 73/159] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:06.099927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:06.125201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e83c3d-657e-47a6-981a-a24e56af3c21 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:06.175921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:06.181539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.182079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.182519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.184600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:06.186225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:06.188125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 87/160] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:06 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:06.194156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:06.260123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84527c3b-1a0e-4f32-8c1f-da5b5f8a58c9 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:06.272778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:06.298401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:06.343240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3840bf-458f-4efe-abd6-de3aeffd5712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:06.343240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3840bf-458f-4efe-abd6-de3aeffd5712 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.343240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3840bf-458f-4efe-abd6-de3aeffd5712 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.343240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3840bf-458f-4efe-abd6-de3aeffd5712 None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.349207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3840bf-458f-4efe-abd6-de3aeffd5712 None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:06.350679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3840bf-458f-4efe-abd6-de3aeffd5712 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:06.352805 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 74/161] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:06 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:06.354117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8e116b-c81b-4c23-bee4-8234936f4458 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.354407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8e116b-c81b-4c23-bee4-8234936f4458 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.354686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8e116b-c81b-4c23-bee4-8234936f4458 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.356100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 75/162] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:06.357784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8e116b-c81b-4c23-bee4-8234936f4458 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:06.372311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8473944-54e5-45b9-8db5-2d4a37c079f9 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:06.422589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1199b0b-b2a1-457b-9316-e3c462c3871b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:06.423530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1199b0b-b2a1-457b-9316-e3c462c3871b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.423978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1199b0b-b2a1-457b-9316-e3c462c3871b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.424415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1199b0b-b2a1-457b-9316-e3c462c3871b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.425409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1199b0b-b2a1-457b-9316-e3c462c3871b None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=02951879c7ac4670a6f8f5177df11cb3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:06.425868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1199b0b-b2a1-457b-9316-e3c462c3871b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:06.427483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1199b0b-b2a1-457b-9316-e3c462c3871b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:06.438807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 88/163] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:06 2022] GET /identity/v3/role_assignments?user.id=02951879c7ac4670a6f8f5177df11cb3&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 608 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:06.440502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e856731b-27f1-4637-849d-5f37b0de4531 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.441011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e856731b-27f1-4637-849d-5f37b0de4531 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.441501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e856731b-27f1-4637-849d-5f37b0de4531 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.472928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e856731b-27f1-4637-849d-5f37b0de4531 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:06.474492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e856731b-27f1-4637-849d-5f37b0de4531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:06.479781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8e116b-c81b-4c23-bee4-8234936f4458 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:06.527206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:06.527866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.528145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.528436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.620364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 89/164] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:06.622762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cae7890-8387-455f-ae14-ca829ab506e5 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.625350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cae7890-8387-455f-ae14-ca829ab506e5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.625925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cae7890-8387-455f-ae14-ca829ab506e5 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.627625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 90/165] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:06.632449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:06.633986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00051533-ba50-43ad-b38e-6e858cdab407 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:06.635276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00051533-ba50-43ad-b38e-6e858cdab407 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:06.635904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00051533-ba50-43ad-b38e-6e858cdab407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:06.662312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-00051533-ba50-43ad-b38e-6e858cdab407 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:06.663723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-00051533-ba50-43ad-b38e-6e858cdab407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:06.720820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:06.737323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:06.739550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:06.754597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 91/166] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:06.755867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00051533-ba50-43ad-b38e-6e858cdab407 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:06.873806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 76/167] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 516 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:06.874962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:06.928747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00051533-ba50-43ad-b38e-6e858cdab407 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:06.958005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:07.001815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.002626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.002914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.003212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.015491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72daed08-305a-447b-8579-593e04360cd8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.028055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:07.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:07.037146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:07.037909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 92/168] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:07.038535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.090556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.091387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.091671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.091948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.110541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:07.118763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:07.119619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:07.120373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:07.122271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 77/169] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:06 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:07.125411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.133595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:07.242765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38fee38-1f37-4ec1-bbb7-7bc8a22b8780 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.275792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47113196-2654-4fa8-899f-aa2c9b09eafc None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.307361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:07.314252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.315161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.315591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.316015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.316144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.317081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:07.317193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.317556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.317764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:07.318005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] PATH_INFO: `/v3/projects/1c57194cea024b6e8fdc9a2d5910c313` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.319485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:07.320308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] RBAC: Authorizing `identity:get_project(project_id=1c57194cea024b6e8fdc9a2d5910c313)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:07.321878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:07.323698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 93/170] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:07 2022] GET /identity/v3/projects/1c57194cea024b6e8fdc9a2d5910c313 => generated 312 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:07.329354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.330410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 78/171] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:07.331440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.438401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00fbefda-56c7-4be3-9824-f60064491999 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.474294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.475208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.475641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.476076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.477097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=1c57194cea024b6e8fdc9a2d5910c313, user.id=02951879c7ac4670a6f8f5177df11cb3) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:07.477555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:07.479162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:07.486352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 79/172] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:07 2022] GET /identity/v3/role_assignments?user.id=02951879c7ac4670a6f8f5177df11cb3&scope.project.id=1c57194cea024b6e8fdc9a2d5910c313&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:07.487297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.552895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:07.598669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd911c77-c33d-4c29-a022-86fd2660ace2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.600240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e8a69-1979-4a22-86a4-cfd7f08e15c2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.618484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:07.640653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.641815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.642357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.642781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] PATH_INFO: `/v3/users/6ccea72aef8d4923b23c0da0e9bf8a5f` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.644235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] RBAC: Authorizing `identity:get_user(user_id=6ccea72aef8d4923b23c0da0e9bf8a5f)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:07.645907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:07.647906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 80/173] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:07 2022] GET /identity/v3/users/6ccea72aef8d4923b23c0da0e9bf8a5f => generated 289 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:07.654325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.695119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.696059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.696522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.696947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.697856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:07.698312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:07.700536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:07.716838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 94/174] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:07.722548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.816570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0dc40d-e5c3-482b-8816-1be957cc965f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.869823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:07.889178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74c1f98-04f7-4906-ba74-e13ce2c02e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.890543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c1f98-04f7-4906-ba74-e13ce2c02e0e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.892174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c1f98-04f7-4906-ba74-e13ce2c02e0e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.892765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74c1f98-04f7-4906-ba74-e13ce2c02e0e None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.894907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74c1f98-04f7-4906-ba74-e13ce2c02e0e None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:07.897868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74c1f98-04f7-4906-ba74-e13ce2c02e0e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:07.899931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 81/175] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:07 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:07.902778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3479808d-aa81-45e4-8b9d-11e09c857fb6 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.903982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3479808d-aa81-45e4-8b9d-11e09c857fb6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.905845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3479808d-aa81-45e4-8b9d-11e09c857fb6 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.908123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 82/176] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:07.911799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3479808d-aa81-45e4-8b9d-11e09c857fb6 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:07.937083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b81ab8e-1deb-4458-ae64-8e9769d0578b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:07.993287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d88b55a-ef28-498c-a2e2-58bd06734ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:07.994570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d88b55a-ef28-498c-a2e2-58bd06734ba8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:07.995098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d88b55a-ef28-498c-a2e2-58bd06734ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:07.995658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d88b55a-ef28-498c-a2e2-58bd06734ba8 None admin] PATH_INFO: `/v3/users/02951879c7ac4670a6f8f5177df11cb3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:07.999327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d88b55a-ef28-498c-a2e2-58bd06734ba8 None admin] RBAC: Authorizing `identity:get_user(user_id=02951879c7ac4670a6f8f5177df11cb3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:07.999327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d88b55a-ef28-498c-a2e2-58bd06734ba8 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:08.001731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 95/177] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:07 2022] GET /identity/v3/users/02951879c7ac4670a6f8f5177df11cb3 => generated 275 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:08.003128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616b3033-d3fd-4885-88a7-05bfd69f6063 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.005289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616b3033-d3fd-4885-88a7-05bfd69f6063 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.005781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616b3033-d3fd-4885-88a7-05bfd69f6063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.031531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-616b3033-d3fd-4885-88a7-05bfd69f6063 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:08.033349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-616b3033-d3fd-4885-88a7-05bfd69f6063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:08.081067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:08.105727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3479808d-aa81-45e4-8b9d-11e09c857fb6 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:08.158069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 96/178] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:08.159514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616b3033-d3fd-4885-88a7-05bfd69f6063 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:08.184765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:08.186061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.186606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.188182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8/users/6ccea72aef8d4923b23c0da0e9bf8a5f/roles/bf3b74a9a9db444a8288ca37ccf736b7` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.196438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] RBAC: Authorizing `identity:create_grant(project_id=68b3ea181434496b91ea5053896483e8, user_id=6ccea72aef8d4923b23c0da0e9bf8a5f, role_id=bf3b74a9a9db444a8288ca37ccf736b7)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:08.196438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:08.218950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 83/179] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:07 2022] PUT /identity/v3/projects/68b3ea181434496b91ea5053896483e8/users/6ccea72aef8d4923b23c0da0e9bf8a5f/roles/bf3b74a9a9db444a8288ca37ccf736b7 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:08.219939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:08.341911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:08.400802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616b3033-d3fd-4885-88a7-05bfd69f6063 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:08.424789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5539e0a-74ab-4ed4-904d-b97ef9803751 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:08.446647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2952a94d-4821-4588-9092-3e170329c51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:08.448188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2952a94d-4821-4588-9092-3e170329c51d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.448686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2952a94d-4821-4588-9092-3e170329c51d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.449776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2952a94d-4821-4588-9092-3e170329c51d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.469356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba6346f-f339-4c01-a7fb-0c57f1646354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:08.470408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba6346f-f339-4c01-a7fb-0c57f1646354 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.470700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba6346f-f339-4c01-a7fb-0c57f1646354 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.471005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba6346f-f339-4c01-a7fb-0c57f1646354 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.485501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba6346f-f339-4c01-a7fb-0c57f1646354 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:08.488767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-eba6346f-f339-4c01-a7fb-0c57f1646354 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:08.494379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:08.495233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:08.496636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 84/180] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:08.498217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474e72d-a2d6-4d29-b9d0-419eea9d00c5 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.498217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474e72d-a2d6-4d29-b9d0-419eea9d00c5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.498217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f474e72d-a2d6-4d29-b9d0-419eea9d00c5 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.503258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 85/181] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:08.503258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c34433-77b7-4e52-99da-a6d9b5cdb7b0 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.503258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c34433-77b7-4e52-99da-a6d9b5cdb7b0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.503258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c34433-77b7-4e52-99da-a6d9b5cdb7b0 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.504572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 86/182] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:08.506035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c34433-77b7-4e52-99da-a6d9b5cdb7b0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:08.532744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2952a94d-4821-4588-9092-3e170329c51d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:08.534830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2952a94d-4821-4588-9092-3e170329c51d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:08.616139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c34433-77b7-4e52-99da-a6d9b5cdb7b0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:08.630530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:08.644808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 97/183] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:08.650594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bd54ae-3581-4bb0-9685-3b4b84cbfe96 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.650887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bd54ae-3581-4bb0-9685-3b4b84cbfe96 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.651165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bd54ae-3581-4bb0-9685-3b4b84cbfe96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.653442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f54e7f-6a96-46b1-a11d-d0cfc872882e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:08.654111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f54e7f-6a96-46b1-a11d-d0cfc872882e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.654395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f54e7f-6a96-46b1-a11d-d0cfc872882e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.654671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f54e7f-6a96-46b1-a11d-d0cfc872882e None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.655431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f54e7f-6a96-46b1-a11d-d0cfc872882e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:08.655743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f54e7f-6a96-46b1-a11d-d0cfc872882e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:08.657244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f54e7f-6a96-46b1-a11d-d0cfc872882e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:08.665720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 87/184] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:08.668883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-50bd54ae-3581-4bb0-9685-3b4b84cbfe96 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:08.669954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bed0c9-0ee3-41f8-91e9-3e28a8530523 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.670341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bed0c9-0ee3-41f8-91e9-3e28a8530523 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.670674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-50bd54ae-3581-4bb0-9685-3b4b84cbfe96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:08.670786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bed0c9-0ee3-41f8-91e9-3e28a8530523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.691509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-70bed0c9-0ee3-41f8-91e9-3e28a8530523 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:08.693239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-70bed0c9-0ee3-41f8-91e9-3e28a8530523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:08.736432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:08.748559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 98/185] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:08.750382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bd54ae-3581-4bb0-9685-3b4b84cbfe96 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:08.773695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 88/186] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:08.774603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bed0c9-0ee3-41f8-91e9-3e28a8530523 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:08.864421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50bd54ae-3581-4bb0-9685-3b4b84cbfe96 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:08.889899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bed0c9-0ee3-41f8-91e9-3e28a8530523 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:08.898487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:08.903799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:08.904875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.905301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.905743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.907081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:08.908585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:08.910621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 99/187] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:08 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:08.911491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:08.933639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:08.934460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:08.934751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:08.935034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] PATH_INFO: `/v3/users/fb9667867ba342789d98e966a79e4609` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:08.949689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] RBAC: Authorizing `identity:get_user(user_id=fb9667867ba342789d98e966a79e4609)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:08.951198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:08.956429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 89/188] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:08 2022] GET /identity/v3/users/fb9667867ba342789d98e966a79e4609 => generated 283 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:08.956429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:08.991587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.028485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af39feee-818c-44bc-863a-5bf5cf9c8af6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.063593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.064508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.064862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.065223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.067997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15897ff8-b678-46da-ae4e-971ad180bfd5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.076278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:09.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:09.089420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:09.090725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 100/189] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:09.092342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.110975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.111942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.112544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.112837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.114051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.115945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:09.117898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 90/190] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:09.122869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.157250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.198852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b6bf04-abeb-4db1-a732-2d2af25fc058 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.226437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.237019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.237683 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.237972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.238197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50f15de-d0bf-4734-8a89-741de56f0b16 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.238320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.239078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] RBAC: Adding query filter params (role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=02951879c7ac4670a6f8f5177df11cb3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:09.239394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.240886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:09.247033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 101/191] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:09 2022] GET /identity/v3/role_assignments?user.id=02951879c7ac4670a6f8f5177df11cb3&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:09.262012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.278296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.279389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.279931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.280433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.366077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.367956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:09.403741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.416694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46317d8d-c9e1-4f8b-892a-dcdf79647f2f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.461149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.462231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.462716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.463208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.464324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:09.465304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.466825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:09.478431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 102/192] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:09.479160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.495808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 91/193] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:09.497491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.543462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.594971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dbdccd-c593-4f4f-94e0-e9dad0aeef28 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.614623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bca691b-4853-495a-983d-dafc22ab2c83 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.631732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.632587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.632884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.633465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.634184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.635627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:09.647900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.655235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.656191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.656528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.656803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] PATH_INFO: `/v3/users/fb9667867ba342789d98e966a79e4609` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.657995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] RBAC: Authorizing `identity:get_user(user_id=fb9667867ba342789d98e966a79e4609)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.659395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 103/194] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:09 2022] POST /identity/v3/users => generated 297 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:09.659632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:09.661626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 92/195] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:09 2022] GET /identity/v3/users/fb9667867ba342789d98e966a79e4609 => generated 283 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:09.665511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.667368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.785709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f269aa3-cc3c-40ae-8535-05c3f9cd53fe None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.821852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fb981f-3396-4864-9bc8-15002ee26be1 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:09.828491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.829111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.829400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.829680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.832639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.864889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:09.865777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:09.865996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:09.866290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:09.867482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.868974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:09.871479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 104/196] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:09 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:09.877231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:09.931839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:09.935578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:09.938294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:10.020140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0853de36-b204-4696-a683-e4f72255dec0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:10.064522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:10.065252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:10.065484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:10.065770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:10.066619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=fb9667867ba342789d98e966a79e4609) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:10.067489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:10.068999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:10.078743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 105/197] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:09 2022] GET /identity/v3/role_assignments?user.id=fb9667867ba342789d98e966a79e4609&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:10.088222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 93/198] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:10.095077 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:10.145886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:10.213881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7603ab15-39db-4470-9de4-fe94401c5fcb None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:10.255704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae9be2a-c634-4b00-ae06-d69780d4859f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:10.256672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae9be2a-c634-4b00-ae06-d69780d4859f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:10.256960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae9be2a-c634-4b00-ae06-d69780d4859f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:10.257234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae9be2a-c634-4b00-ae06-d69780d4859f None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:10.261276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae9be2a-c634-4b00-ae06-d69780d4859f None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:10.262960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae9be2a-c634-4b00-ae06-d69780d4859f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:10.265424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 106/199] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:10 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:10.271321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:10.318287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d86c1-8e9c-4c6a-b01f-88112b84b624 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:10.318593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d86c1-8e9c-4c6a-b01f-88112b84b624 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:10.318856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d86c1-8e9c-4c6a-b01f-88112b84b624 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:10.320554 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 107/200] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:57:10.325781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c53025-c1ec-4a1a-8431-e0cdc419fcc4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:10.326203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c53025-c1ec-4a1a-8431-e0cdc419fcc4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:10.326582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c53025-c1ec-4a1a-8431-e0cdc419fcc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:10.361455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d9c53025-c1ec-4a1a-8431-e0cdc419fcc4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:10.368782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d9c53025-c1ec-4a1a-8431-e0cdc419fcc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:10.414617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:10.420603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca88e55-7b62-4275-a305-89d0ce8c5a95 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:10.462626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:10.463670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:10.464251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:10.464817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:10.465902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] RBAC: Adding query filter params (role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=fb9667867ba342789d98e966a79e4609) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:10.466407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:10.468760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:10.475161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 94/201] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:10 2022] GET /identity/v3/role_assignments?user.id=fb9667867ba342789d98e966a79e4609&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 259 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:10.490033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 108/202] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:10.525729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:10.658842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42adceda-bc69-43d1-b4fd-cce2ed44edf2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:10.697522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:10.698875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:10.699356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:10.699820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:10.710895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:10.719699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:10.720619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:10.721483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:10.723864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 95/203] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:10 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:10.729769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c53025-c1ec-4a1a-8431-e0cdc419fcc4 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:10.787137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:10.865942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c53025-c1ec-4a1a-8431-e0cdc419fcc4 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:10.907490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:10.908180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:10.908497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:10.908778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:10.909547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] RBAC: Adding query filter params (name=reader) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:10.909856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:10.911244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:10.921314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 109/204] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:10.934545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:10.996931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:11.121948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc33df-b2b0-4532-9fa7-4245b097d997 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:11.171895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:11.172806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:11.173087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:11.173362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] PATH_INFO: `/v3/users/6ccea72aef8d4923b23c0da0e9bf8a5f` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:11.177125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] RBAC: Authorizing `identity:get_user(user_id=6ccea72aef8d4923b23c0da0e9bf8a5f)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:11.178585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:11.180222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 96/205] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:10 2022] GET /identity/v3/users/6ccea72aef8d4923b23c0da0e9bf8a5f => generated 289 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:11.186304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:11.340948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:11.347104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97da13c0-3de9-4ba3-b2b2-20d756887ecf None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:11.392028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:11.392738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:11.393031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:11.393311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:11.483037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:11.484534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:11.612918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:11.619292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 110/206] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:11.624838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:11.771280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:11.781456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2724279a-3298-4ea6-9271-0da499e141b2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:11.834244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7023abab-4303-4e44-a86a-fea3e23d4a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:11.835124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023abab-4303-4e44-a86a-fea3e23d4a99 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:11.835411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023abab-4303-4e44-a86a-fea3e23d4a99 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:11.835687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7023abab-4303-4e44-a86a-fea3e23d4a99 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:11.841108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7023abab-4303-4e44-a86a-fea3e23d4a99 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:11.842595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7023abab-4303-4e44-a86a-fea3e23d4a99 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:11.844670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 97/207] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:11 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:11.850020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:11.968712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0af12e-c6f9-4873-a37c-80135fbec806 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:12.015683 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecdf77d-5d71-4268-a669-e01e6cae2dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:12.016353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdf77d-5d71-4268-a669-e01e6cae2dd2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.016665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdf77d-5d71-4268-a669-e01e6cae2dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.016945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecdf77d-5d71-4268-a669-e01e6cae2dd2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.017793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdf77d-5d71-4268-a669-e01e6cae2dd2 None admin] RBAC: Adding query filter params (role.id=bf3b74a9a9db444a8288ca37ccf736b7, scope.project.id=68b3ea181434496b91ea5053896483e8, user.id=6ccea72aef8d4923b23c0da0e9bf8a5f) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:12.018187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdf77d-5d71-4268-a669-e01e6cae2dd2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:12.021499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecdf77d-5d71-4268-a669-e01e6cae2dd2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:12.026651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 111/208] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:11 2022] GET /identity/v3/role_assignments?user.id=6ccea72aef8d4923b23c0da0e9bf8a5f&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=bf3b74a9a9db444a8288ca37ccf736b7 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:12.152516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155881bf-2057-4bd8-a7e1-e1242ab9df68 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.153064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155881bf-2057-4bd8-a7e1-e1242ab9df68 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.153064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155881bf-2057-4bd8-a7e1-e1242ab9df68 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.154854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 98/209] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:12.160049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e7f56-6de1-4a08-8ece-8468d49ab0a7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.160343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e7f56-6de1-4a08-8ece-8468d49ab0a7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.160649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890e7f56-6de1-4a08-8ece-8468d49ab0a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.182094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-890e7f56-6de1-4a08-8ece-8468d49ab0a7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:12.183790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-890e7f56-6de1-4a08-8ece-8468d49ab0a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:12.263192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:12.274564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 112/210] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:12.297938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155881bf-2057-4bd8-a7e1-e1242ab9df68 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:12.445301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155881bf-2057-4bd8-a7e1-e1242ab9df68 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:12.488262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9e3e01-fb60-4f60-a6a2-5ec46d511e0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:12.489154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9e3e01-fb60-4f60-a6a2-5ec46d511e0a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.489399 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9e3e01-fb60-4f60-a6a2-5ec46d511e0a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.489677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9e3e01-fb60-4f60-a6a2-5ec46d511e0a None admin] PATH_INFO: `/v3/roles/member` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.505253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9e3e01-fb60-4f60-a6a2-5ec46d511e0a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-7a9e3e01-fb60-4f60-a6a2-5ec46d511e0a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:12.513745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:12.514623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:12.515647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 99/211] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:12.522501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e7f56-6de1-4a08-8ece-8468d49ab0a7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:12.599075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6dc7d6-bc82-4386-8b66-6025ab6d0897 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.599417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6dc7d6-bc82-4386-8b66-6025ab6d0897 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.599633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6dc7d6-bc82-4386-8b66-6025ab6d0897 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.601099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 100/212] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:12.606694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c882e72-5310-40b4-bc25-424b8b6ff846 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.606916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c882e72-5310-40b4-bc25-424b8b6ff846 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.607192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c882e72-5310-40b4-bc25-424b8b6ff846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.624565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7c882e72-5310-40b4-bc25-424b8b6ff846 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:12.625762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7c882e72-5310-40b4-bc25-424b8b6ff846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:12.632283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890e7f56-6de1-4a08-8ece-8468d49ab0a7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:12.641593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:12.671625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3ddb37-cacb-47f2-b6da-0388f616d792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:12.672307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3ddb37-cacb-47f2-b6da-0388f616d792 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.672639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3ddb37-cacb-47f2-b6da-0388f616d792 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.672925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3ddb37-cacb-47f2-b6da-0388f616d792 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.674063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3ddb37-cacb-47f2-b6da-0388f616d792 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:12.674385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3ddb37-cacb-47f2-b6da-0388f616d792 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:12.676141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3ddb37-cacb-47f2-b6da-0388f616d792 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:12.685360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 113/213] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:12.685360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b612acf-2b13-43f9-b210-0a7cc3ebf363 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.685360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b612acf-2b13-43f9-b210-0a7cc3ebf363 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.685360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b612acf-2b13-43f9-b210-0a7cc3ebf363 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.685360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 114/214] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:12 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:12.690947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b612acf-2b13-43f9-b210-0a7cc3ebf363 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:12.703048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 101/215] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:12.704255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd402d23-0e4c-454e-accf-8548115f3a72 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.704573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd402d23-0e4c-454e-accf-8548115f3a72 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.704850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd402d23-0e4c-454e-accf-8548115f3a72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.728551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-dd402d23-0e4c-454e-accf-8548115f3a72 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:12.729878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-dd402d23-0e4c-454e-accf-8548115f3a72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:12.752694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:12.804165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 102/216] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:12.812562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd402d23-0e4c-454e-accf-8548115f3a72 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:12.812647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b612acf-2b13-43f9-b210-0a7cc3ebf363 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:12.848911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:12.849875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.850160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.850435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] PATH_INFO: `/v3/users/6d19713fb78f4a2d9dcac5198356bc26` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.865851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] RBAC: Authorizing `identity:get_user(user_id=6d19713fb78f4a2d9dcac5198356bc26)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:12.867305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:12.869167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 115/217] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:12 2022] GET /identity/v3/users/6d19713fb78f4a2d9dcac5198356bc26 => generated 297 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:12.873521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:12.900279 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:12.926282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd402d23-0e4c-454e-accf-8548115f3a72 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:12.963416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:12.964145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:12.965050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:12.965302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:12.966423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:12.967877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:12.972557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 103/218] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:12.972557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:12.989076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba3766-a57b-4d49-8e1b-467ac5c29d5d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.003379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:13.026719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.028075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.028353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.028677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.039319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:13.047498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:13.048655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:13.049590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 116/219] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:13.050493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.086354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c452932-a97f-4b4a-8d6d-9802f588632d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.120111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4480f10-e12e-4550-84af-1f1d5da8d673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.121360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4480f10-e12e-4550-84af-1f1d5da8d673 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.121631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4480f10-e12e-4550-84af-1f1d5da8d673 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.121908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4480f10-e12e-4550-84af-1f1d5da8d673 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.139450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:13.149066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac42254-3c27-47a1-ac45-3a63f63863a5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.183826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bbdfc7-2198-47b6-81ee-8dcd67ca5b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.184521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bbdfc7-2198-47b6-81ee-8dcd67ca5b56 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.184810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bbdfc7-2198-47b6-81ee-8dcd67ca5b56 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.185089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bbdfc7-2198-47b6-81ee-8dcd67ca5b56 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.185798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bbdfc7-2198-47b6-81ee-8dcd67ca5b56 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:13.187264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bbdfc7-2198-47b6-81ee-8dcd67ca5b56 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:13.190114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4480f10-e12e-4550-84af-1f1d5da8d673 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:13.191577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4480f10-e12e-4550-84af-1f1d5da8d673 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:13.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 117/220] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:13 2022] POST /identity/v3/users => generated 297 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:13.208749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54f829-fd9a-41e6-a93c-d0c1865086eb None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.209043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54f829-fd9a-41e6-a93c-d0c1865086eb None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.209347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54f829-fd9a-41e6-a93c-d0c1865086eb None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.210778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 118/221] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:13.211348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd54f829-fd9a-41e6-a93c-d0c1865086eb None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.251881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:13.315147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 104/222] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:13.315147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0337cc89-2cc3-43a1-ac3d-ed9fd415dc90 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.315147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0337cc89-2cc3-43a1-ac3d-ed9fd415dc90 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.315147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0337cc89-2cc3-43a1-ac3d-ed9fd415dc90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.332962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd54f829-fd9a-41e6-a93c-d0c1865086eb None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.338607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0337cc89-2cc3-43a1-ac3d-ed9fd415dc90 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:13.339928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0337cc89-2cc3-43a1-ac3d-ed9fd415dc90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:13.374543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.374543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.374822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.375032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.376931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:13.376931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:13.377576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:13.386214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 119/223] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:13.387594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.404071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:13.436282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 105/224] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:13.437515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0337cc89-2cc3-43a1-ac3d-ed9fd415dc90 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.499051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:13.510143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daa7855-3310-4d81-b468-dfa70aea783d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.551652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.552567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.552855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.553134 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.554342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:13.555817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:13.557677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 120/225] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:13 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:13.558489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.570275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0337cc89-2cc3-43a1-ac3d-ed9fd415dc90 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.615301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.616481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.616769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.617045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] PATH_INFO: `/v3/users/fb9667867ba342789d98e966a79e4609` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.618493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] RBAC: Authorizing `identity:get_user(user_id=fb9667867ba342789d98e966a79e4609)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:13.620412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:13.626584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 106/226] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:13 2022] GET /identity/v3/users/fb9667867ba342789d98e966a79e4609 => generated 283 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:13.626584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.678209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:13.690052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa4e09-1e66-4d6a-bd61-ddaf758cefa3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.733717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.734911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.735394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.736029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.742710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8539cd5-d9b0-4beb-882d-f3ac1495074d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.747184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:13.755728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:13.756677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:13.757455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:13.759917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 121/227] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:13.761060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.786060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:13.792558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:13.793367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:13.793648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:13.793929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:13.795025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=6d19713fb78f4a2d9dcac5198356bc26) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:13.795345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:13.797500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:13.807112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 107/228] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:13 2022] GET /identity/v3/role_assignments?user.id=6d19713fb78f4a2d9dcac5198356bc26&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:13.811056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:13.958184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4948b7e8-26af-4cdc-98b1-fa375e96239f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.970070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d2d651-d09f-46c9-8fcd-86cb30c7d615 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:13.989219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:14.008597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:14.009979 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:14.010623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:14.011204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:14.012758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:14.014664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:14.017801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 122/229] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:13 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:14.021286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:14.021973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:14.022262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:14.022541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:14.023402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:14.024144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:14.025159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:14.025708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:14.043796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 108/230] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:14.049877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:14.190919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:14.215152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acad6631-f279-497d-99cd-bb64f1d12f1e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:14.227212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf4f6e5-835d-4e96-9401-e19b8120182f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:14.266761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:14.267585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:14.267871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:14.268160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/fb9667867ba342789d98e966a79e4609/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:14.270368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:14.271478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:14.272008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:14.272511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] PATH_INFO: `/v3/users/fb9667867ba342789d98e966a79e4609` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:14.273690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] RBAC: Authorizing `identity:create_grant(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user_id=fb9667867ba342789d98e966a79e4609, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:14.274316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] RBAC: Authorizing `identity:get_user(user_id=fb9667867ba342789d98e966a79e4609)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:14.275157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:14.276231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:14.278186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 109/231] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:14 2022] GET /identity/v3/users/fb9667867ba342789d98e966a79e4609 => generated 283 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:14.285916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:14.303458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 123/232] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:14 2022] PUT /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/fb9667867ba342789d98e966a79e4609/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 09:57:14.344975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:14.429932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cf013a-a44c-4f9b-a44a-6906f16405b0 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:14.482430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1128e759-e869-4f67-b374-e702d2bf0398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:14.483293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1128e759-e869-4f67-b374-e702d2bf0398 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:14.483578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1128e759-e869-4f67-b374-e702d2bf0398 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:14.483851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1128e759-e869-4f67-b374-e702d2bf0398 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:14.488817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1128e759-e869-4f67-b374-e702d2bf0398 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:14.490516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1128e759-e869-4f67-b374-e702d2bf0398 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:14.493002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 110/233] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:14 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:14.498398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:14.613350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db02d480-853c-4330-bb19-8aa3d0375689 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:14.654799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce455ff5-b513-4ac9-9a48-3ec7347b7f2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:14.655762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce455ff5-b513-4ac9-9a48-3ec7347b7f2f None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:14.656049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce455ff5-b513-4ac9-9a48-3ec7347b7f2f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:14.656362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce455ff5-b513-4ac9-9a48-3ec7347b7f2f None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/fb9667867ba342789d98e966a79e4609/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:14.662225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce455ff5-b513-4ac9-9a48-3ec7347b7f2f None admin] RBAC: Authorizing `identity:create_grant(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user_id=fb9667867ba342789d98e966a79e4609, role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:14.663797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce455ff5-b513-4ac9-9a48-3ec7347b7f2f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:14.683339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 124/234] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:14 2022] PUT /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/fb9667867ba342789d98e966a79e4609/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:15.707084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd8e701-dc52-4467-81d6-4312135e6061 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:15.707622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd8e701-dc52-4467-81d6-4312135e6061 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:15.707622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd8e701-dc52-4467-81d6-4312135e6061 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:15.709263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 111/235] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:15.714534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc63e7ae-4a32-4c19-8e8b-0faf705d0299 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:15.714926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc63e7ae-4a32-4c19-8e8b-0faf705d0299 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:15.715108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc63e7ae-4a32-4c19-8e8b-0faf705d0299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:15.736934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-bc63e7ae-4a32-4c19-8e8b-0faf705d0299 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:15.738242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bc63e7ae-4a32-4c19-8e8b-0faf705d0299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:15.740563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:15.831081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 125/236] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:15.866208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd8e701-dc52-4467-81d6-4312135e6061 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:15.913825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571512b5-1496-4696-ae80-028840fde2ee None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:15.914132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571512b5-1496-4696-ae80-028840fde2ee None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:15.914396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571512b5-1496-4696-ae80-028840fde2ee None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:15.916309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 126/237] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:15.929614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc495c8e-4d84-46f5-84cb-7e7f88b7b768 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:15.929908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc495c8e-4d84-46f5-84cb-7e7f88b7b768 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:15.930194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc495c8e-4d84-46f5-84cb-7e7f88b7b768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:15.962705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-cc495c8e-4d84-46f5-84cb-7e7f88b7b768 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:15.963823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cc495c8e-4d84-46f5-84cb-7e7f88b7b768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:16.023643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd8e701-dc52-4467-81d6-4312135e6061 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:16.039104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:16.091181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:16.091986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.092264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.092562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.101121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 127/238] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:16.117393 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:16.125741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:16.127800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:16.129769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:16.136435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 112/239] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:16.138360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc495c8e-4d84-46f5-84cb-7e7f88b7b768 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:16.142233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:16.211802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:16.254552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc495c8e-4d84-46f5-84cb-7e7f88b7b768 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:16.295378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:16.296057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.296341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.296647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.297408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] RBAC: Adding query filter params (name=reader) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:16.297716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:16.299118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:16.317107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 128/240] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:16.325389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:16.354252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878273f3-01e9-44e8-8808-ce78b6b7f4eb None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:16.394527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a8dd73-3581-49d0-a4dd-dbfcdbb3e689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:16.395428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a8dd73-3581-49d0-a4dd-dbfcdbb3e689 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.395766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a8dd73-3581-49d0-a4dd-dbfcdbb3e689 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.396047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a8dd73-3581-49d0-a4dd-dbfcdbb3e689 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.397259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a8dd73-3581-49d0-a4dd-dbfcdbb3e689 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:16.398851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a8dd73-3581-49d0-a4dd-dbfcdbb3e689 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:16.400920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 113/241] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:16.401979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c5c0b-b461-41c0-a64b-af7f9cb2c228 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.402267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c5c0b-b461-41c0-a64b-af7f9cb2c228 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.402547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c5c0b-b461-41c0-a64b-af7f9cb2c228 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.407910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 114/242] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:16.409139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca430f6-9bef-4434-8cf5-f286661b8e5d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.409428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca430f6-9bef-4434-8cf5-f286661b8e5d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.409714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca430f6-9bef-4434-8cf5-f286661b8e5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.426866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5ca430f6-9bef-4434-8cf5-f286661b8e5d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:16.428056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5ca430f6-9bef-4434-8cf5-f286661b8e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:16.438961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626dea80-8572-4373-8183-f3468c60c06e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:16.460486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:16.469969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:16.476096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:16.477115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.477404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.477682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] PATH_INFO: `/v3/users/117f0bab037044c6a6cb3ada31c08221` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.493624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] RBAC: Authorizing `identity:get_user(user_id=117f0bab037044c6a6cb3ada31c08221)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:16.495207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:16.498084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 129/243] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:16 2022] GET /identity/v3/users/117f0bab037044c6a6cb3ada31c08221 => generated 297 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:16.498915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:16.508547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 115/244] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:16.513320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca430f6-9bef-4434-8cf5-f286661b8e5d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:16.606935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acb8abe-b7ec-4af2-8a2b-6e55257ba955 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:16.630536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca430f6-9bef-4434-8cf5-f286661b8e5d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:16.647146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:16.648656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.649114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.649658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.650724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:16.652801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:16.667350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871cd7b3-2ee9-4427-bc6d-40556dc4ca88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:16.668302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871cd7b3-2ee9-4427-bc6d-40556dc4ca88 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.668681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871cd7b3-2ee9-4427-bc6d-40556dc4ca88 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.669009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871cd7b3-2ee9-4427-bc6d-40556dc4ca88 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.674410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 130/245] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:16 2022] POST /identity/v3/users => generated 293 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:16.681552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:16.714690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:16.746280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871cd7b3-2ee9-4427-bc6d-40556dc4ca88 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:16.747817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871cd7b3-2ee9-4427-bc6d-40556dc4ca88 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:16.754394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:16.823280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db080d02-1c93-4c66-9127-a19b40ec38b6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:16.864913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7141a42-3083-4377-8203-f5b04c3f3913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:16.866125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7141a42-3083-4377-8203-f5b04c3f3913 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.866612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7141a42-3083-4377-8203-f5b04c3f3913 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.867085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7141a42-3083-4377-8203-f5b04c3f3913 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.878902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7141a42-3083-4377-8203-f5b04c3f3913 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d7141a42-3083-4377-8203-f5b04c3f3913 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:16.887671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:16.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:16.889356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:16.891508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 131/246] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:16.891508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169567d-a515-46fa-93c2-ac4ff8b80612 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.891632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169567d-a515-46fa-93c2-ac4ff8b80612 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.891852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169567d-a515-46fa-93c2-ac4ff8b80612 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.897863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 132/247] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:16 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:16.898626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169567d-a515-46fa-93c2-ac4ff8b80612 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:16.904991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 116/248] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:16.906657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50b2589-ef37-472d-89ae-de646a8446bd None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:16.907424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50b2589-ef37-472d-89ae-de646a8446bd None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:16.908918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50b2589-ef37-472d-89ae-de646a8446bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:16.933707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a50b2589-ef37-472d-89ae-de646a8446bd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:16.933707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a50b2589-ef37-472d-89ae-de646a8446bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:17.001843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:17.010494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 117/249] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:17.012635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50b2589-ef37-472d-89ae-de646a8446bd None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.053921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169567d-a515-46fa-93c2-ac4ff8b80612 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.056533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:17.097487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.098089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.098377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.098652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.099414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:17.099729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:17.101169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:17.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 133/250] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:17.111733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.136186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50b2589-ef37-472d-89ae-de646a8446bd None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.179965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.180880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.181158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.182037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.182618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:17.184421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:17.186078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 118/251] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:17.186970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.250447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb28ae7-31db-4ad1-8715-71d5a8a27c71 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.263518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:17.297821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b294c7-8239-4f18-9380-33a8d1251299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.298893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b294c7-8239-4f18-9380-33a8d1251299 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.299180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b294c7-8239-4f18-9380-33a8d1251299 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.299707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b294c7-8239-4f18-9380-33a8d1251299 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.306438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc81330-7b0c-4d5d-8553-d24b5b3febc1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.313818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b294c7-8239-4f18-9380-33a8d1251299 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b9b294c7-8239-4f18-9380-33a8d1251299 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:17.324795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:17.325677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:17.326470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:17.327066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 134/252] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:17.328348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5443a494-5b4d-419b-9dd4-b89ee96a2641 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.328622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5443a494-5b4d-419b-9dd4-b89ee96a2641 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.328898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5443a494-5b4d-419b-9dd4-b89ee96a2641 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.330173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 135/253] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:17.330870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5443a494-5b4d-419b-9dd4-b89ee96a2641 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.351094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.352011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.352421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.352780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] PATH_INFO: `/v3/users/6d19713fb78f4a2d9dcac5198356bc26` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.354110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] RBAC: Authorizing `identity:get_user(user_id=6d19713fb78f4a2d9dcac5198356bc26)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:17.355684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:17.357828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 119/254] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/users/6d19713fb78f4a2d9dcac5198356bc26 => generated 297 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:17.358701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.369420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:17.465712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d80d3e-286e-4cfc-bf63-9babe18a08be None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.468956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5443a494-5b4d-419b-9dd4-b89ee96a2641 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.503305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6369666-6db5-4691-9c7a-6e8045a2e605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.503989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6369666-6db5-4691-9c7a-6e8045a2e605 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.504323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6369666-6db5-4691-9c7a-6e8045a2e605 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.504641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6369666-6db5-4691-9c7a-6e8045a2e605 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.505456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6369666-6db5-4691-9c7a-6e8045a2e605 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:17.505768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6369666-6db5-4691-9c7a-6e8045a2e605 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:17.507224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6369666-6db5-4691-9c7a-6e8045a2e605 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:17.508769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:17.516976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 120/255] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:17.517580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.518458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.518932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.519418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.520491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] RBAC: Adding query filter params (role.id=bf3b74a9a9db444a8288ca37ccf736b7, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=117f0bab037044c6a6cb3ada31c08221) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:17.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:17.522713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:17.523030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63744353-69f5-48a6-8651-efbd8343224c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.523350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63744353-69f5-48a6-8651-efbd8343224c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.523643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63744353-69f5-48a6-8651-efbd8343224c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.530945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 136/256] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/role_assignments?user.id=117f0bab037044c6a6cb3ada31c08221&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=bf3b74a9a9db444a8288ca37ccf736b7 => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:17.536433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.548935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-63744353-69f5-48a6-8651-efbd8343224c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:17.552438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-63744353-69f5-48a6-8651-efbd8343224c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:17.662380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 121/257] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:17.664863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63744353-69f5-48a6-8651-efbd8343224c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.752809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:17.756277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7901bb5-e18d-461d-8f95-da1d0c5808fb None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.800207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.801136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.801514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.801886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.803213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:17.804806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:17.807428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 137/258] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:17.809192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.862661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63744353-69f5-48a6-8651-efbd8343224c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.912835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:17.920787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f960b6a6-92d1-42ad-aec6-8d8b4f57952c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:17.948573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.949694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.949985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.950266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] PATH_INFO: `/v3/users/fb9667867ba342789d98e966a79e4609` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.956775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:17.957257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] RBAC: Authorizing `identity:get_user(user_id=fb9667867ba342789d98e966a79e4609)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:17.957862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:17.958242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:17.958613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:17.958845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:17.960963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 122/259] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/users/fb9667867ba342789d98e966a79e4609 => generated 283 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:17.961840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:17.970991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:17.979668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:17.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:17.981991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 138/260] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:17.982914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:18.013457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:18.101875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6cb609-3807-419a-862a-cea5d046e985 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:18.170619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:18.171301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:18.171684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:18.172059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:18.188148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d93c1f-941c-4198-9d01-45b1b5f29cd4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:18.231330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:18.232532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:18.234572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:18.235035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/6d19713fb78f4a2d9dcac5198356bc26/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:18.240305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] RBAC: Authorizing `identity:create_grant(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user_id=6d19713fb78f4a2d9dcac5198356bc26, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:18.242125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:18.270686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 123/261] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:17 2022] PUT /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/6d19713fb78f4a2d9dcac5198356bc26/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:18.271740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:18.309406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:18.311384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:18.352423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:18.433768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:18.516575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 139/262] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:18.527328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:18.555324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1ac800-14d2-410a-80ce-c5cba1717e1f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:18.642005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdafa90-5a0e-4bb2-8bb1-94ae2823a6d4 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:18.676425 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:18.688323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:18.689462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:18.689999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:18.690454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:18.691956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:18.693745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:18.696119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 140/263] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:18 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:18.700412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0806887e-e366-4372-851f-6e327252db74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:18.700412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0806887e-e366-4372-851f-6e327252db74 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:18.700412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0806887e-e366-4372-851f-6e327252db74 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:18.700412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0806887e-e366-4372-851f-6e327252db74 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:18.700746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0806887e-e366-4372-851f-6e327252db74 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:18.700746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0806887e-e366-4372-851f-6e327252db74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:18.701834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0806887e-e366-4372-851f-6e327252db74 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:18.710089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:18.721593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 124/264] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 450 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:18.727521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0806887e-e366-4372-851f-6e327252db74 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:18.891258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591dd8c4-6451-4584-9d7e-c162c25773d4 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:18.909408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0806887e-e366-4372-851f-6e327252db74 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:18.952841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:18.974290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:18.974958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:18.975246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:18.975524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:18.976369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=fb9667867ba342789d98e966a79e4609) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:18.976717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:18.978171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:18.997069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 141/265] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:18 2022] GET /identity/v3/role_assignments?user.id=fb9667867ba342789d98e966a79e4609&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:19.003910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec785ddd-a96e-4946-b666-98d1f8fa526a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:19.004864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec785ddd-a96e-4946-b666-98d1f8fa526a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.005154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec785ddd-a96e-4946-b666-98d1f8fa526a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.005438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec785ddd-a96e-4946-b666-98d1f8fa526a None admin] PATH_INFO: `/v3/users/fb9667867ba342789d98e966a79e4609` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.006769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec785ddd-a96e-4946-b666-98d1f8fa526a None admin] RBAC: Authorizing `identity:get_user(user_id=fb9667867ba342789d98e966a79e4609)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:19.008282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec785ddd-a96e-4946-b666-98d1f8fa526a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:19.010075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 125/266] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:18 2022] GET /identity/v3/users/fb9667867ba342789d98e966a79e4609 => generated 283 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:19.016384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:19.091430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:19.158392 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa67924f-10af-414c-a363-fe0a25efd618 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:19.199735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:19.200427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.200721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.201000 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.222419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeba5e1-7e63-400d-a1f5-e35bf6b628cd None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.222613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeba5e1-7e63-400d-a1f5-e35bf6b628cd None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.222891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeba5e1-7e63-400d-a1f5-e35bf6b628cd None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.224580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 126/267] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:19.229938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d77045-fff6-459d-b6e3-cd0f9c4cf99b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.230781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d77045-fff6-459d-b6e3-cd0f9c4cf99b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.231064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d77045-fff6-459d-b6e3-cd0f9c4cf99b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.268128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-88d77045-fff6-459d-b6e3-cd0f9c4cf99b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:19.270468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-88d77045-fff6-459d-b6e3-cd0f9c4cf99b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:19.355477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:19.357010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:19.391526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 127/268] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:19.393244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfcd5f5-9eb4-4dc6-aa7f-855bf53b2155 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.393752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfcd5f5-9eb4-4dc6-aa7f-855bf53b2155 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.394225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfcd5f5-9eb4-4dc6-aa7f-855bf53b2155 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.395709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 128/269] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:19.401697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e1f7d-56cb-43e6-bfee-818a9a2ac376 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.402765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e1f7d-56cb-43e6-bfee-818a9a2ac376 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.403278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e1f7d-56cb-43e6-bfee-818a9a2ac376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.433657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6c7e1f7d-56cb-43e6-bfee-818a9a2ac376 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:19.435140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6c7e1f7d-56cb-43e6-bfee-818a9a2ac376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:19.455291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:19.487640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 142/270] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 472 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:19.488831 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:19.524605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 129/271] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:19.525995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7e1f7d-56cb-43e6-bfee-818a9a2ac376 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:19.584245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:19.617726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54dd788-3b6b-4213-a7fd-b21c294e4f24 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:19.653261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7e1f7d-56cb-43e6-bfee-818a9a2ac376 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:19.664671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:19.665706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.666125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.666583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.677956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:19.687025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:19.688078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:19.688980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:19.690643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 143/272] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:19.695593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:19.698048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:19.699769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.700253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.700752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.702201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:19.703886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:19.706208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 130/273] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:19 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:19.707216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:19.708131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:19.810258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f746ea07-61b4-44bc-9681-81b55dc0d0d6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:19.813768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dd16db-5112-46e1-84c0-d69ecee8a344 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:19.825626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:19.847083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:19.847884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.848175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.848508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.849765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:19.851223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:19.853061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 144/274] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:19.853672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:19.854104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017727af-f048-4236-b868-668eacc97b54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:19.855032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017727af-f048-4236-b868-668eacc97b54 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.855517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017727af-f048-4236-b868-668eacc97b54 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.856033 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017727af-f048-4236-b868-668eacc97b54 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.857020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017727af-f048-4236-b868-668eacc97b54 None admin] RBAC: Adding query filter params (name=member) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:19.857521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017727af-f048-4236-b868-668eacc97b54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:19.859113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017727af-f048-4236-b868-668eacc97b54 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:19.872752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 131/275] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:19.873935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017727af-f048-4236-b868-668eacc97b54 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:19.936222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:19.960679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b817fd4a-2d01-408c-91da-e9638233a647 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:19.991126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017727af-f048-4236-b868-668eacc97b54 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:19.996931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:19.997603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:19.997893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:19.998173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:19.999013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] RBAC: Adding query filter params (role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=fb9667867ba342789d98e966a79e4609) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:19.999334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:20.001062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:20.007330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 145/276] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:19 2022] GET /identity/v3/role_assignments?user.id=fb9667867ba342789d98e966a79e4609&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:20.008403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:20.032139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa879c3-a65b-44d6-b57a-50ce2a4d73dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:20.033287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa879c3-a65b-44d6-b57a-50ce2a4d73dc None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.033770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa879c3-a65b-44d6-b57a-50ce2a4d73dc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.034244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa879c3-a65b-44d6-b57a-50ce2a4d73dc None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.035353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa879c3-a65b-44d6-b57a-50ce2a4d73dc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:20.037117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa879c3-a65b-44d6-b57a-50ce2a4d73dc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:20.061862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 132/277] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:19 2022] POST /identity/v3/users => generated 293 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:20.142001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8714393-e58f-43b4-85a9-edccf7dff511 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:20.185973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:20.186876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.187167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.187447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] PATH_INFO: `/v3/users/11f055ec66ea43c7a70ce81ca351e99c` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.201093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:20.204700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] RBAC: Authorizing `identity:get_user(user_id=11f055ec66ea43c7a70ce81ca351e99c)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:20.206376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:20.208548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 146/278] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:20 2022] GET /identity/v3/users/11f055ec66ea43c7a70ce81ca351e99c => generated 293 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:20.212354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724665cd-e52f-4c3c-aac7-614fac94b658 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.212744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724665cd-e52f-4c3c-aac7-614fac94b658 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.213268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724665cd-e52f-4c3c-aac7-614fac94b658 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.214084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:20.215236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 133/279] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:20.220978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f680dfd-37b2-48c4-b0ee-04636997188d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.221502 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f680dfd-37b2-48c4-b0ee-04636997188d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.221976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f680dfd-37b2-48c4-b0ee-04636997188d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.245869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0f680dfd-37b2-48c4-b0ee-04636997188d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:20.247957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0f680dfd-37b2-48c4-b0ee-04636997188d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:20.276791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:20.331900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b79d19-fcb8-4c3f-908a-8e532745441c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:20.336531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 134/280] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:20.357068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f680dfd-37b2-48c4-b0ee-04636997188d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:20.374018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:20.374981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.375264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.375532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.376371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, user.id=11f055ec66ea43c7a70ce81ca351e99c, scope.system=all) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:20.376796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:20.378215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:20.388017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 147/281] 10.222.0.20 () {56 vars in 1308 bytes} [Tue Jul 5 09:57:20 2022] GET /identity/v3/role_assignments?user.id=11f055ec66ea43c7a70ce81ca351e99c&scope.system=all&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:20.483798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f680dfd-37b2-48c4-b0ee-04636997188d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:20.527438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:20.529541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.530136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.530744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.545147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:20.553208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:20.554293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:20.555154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:20.556012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 135/282] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:20.562670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:20.632134 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:20.675096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1772505f-213c-45a6-b663-d790cf8cc6de None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:20.715592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:20.716271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.716586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.716862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.717624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] RBAC: Adding query filter params (name=reader) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:20.717932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:20.719313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:20.728503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 148/283] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:20 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:20.734156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:20.757396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:20.867233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0f56f0-265b-4a9d-9b1e-34606dd09a4d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:20.912146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:20.913032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:20.913318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:20.913595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] PATH_INFO: `/v3/users/117f0bab037044c6a6cb3ada31c08221` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:20.914851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] RBAC: Authorizing `identity:get_user(user_id=117f0bab037044c6a6cb3ada31c08221)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:20.916327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:20.918147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 136/284] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:20 2022] GET /identity/v3/users/117f0bab037044c6a6cb3ada31c08221 => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:20.923929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:21.043865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f8117-b08f-45ad-a710-56a08aec4a4c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:21.078745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:21.089078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b3a9fa-24ad-4dd5-9073-809bb51cdf6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:21.090489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3a9fa-24ad-4dd5-9073-809bb51cdf6b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:21.090775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3a9fa-24ad-4dd5-9073-809bb51cdf6b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:21.091051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b3a9fa-24ad-4dd5-9073-809bb51cdf6b None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:21.092245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3a9fa-24ad-4dd5-9073-809bb51cdf6b None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:21.093734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b3a9fa-24ad-4dd5-9073-809bb51cdf6b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:21.095454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 149/285] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:20 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:21.101170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:21.182646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48adf9-743a-48da-a042-c389034e3fcc None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:21.182968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48adf9-743a-48da-a042-c389034e3fcc None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:21.183253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be48adf9-743a-48da-a042-c389034e3fcc None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:21.184982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 150/286] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:57:21.190726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc8ec7f-5e10-49f0-92a2-c28c7cd9c440 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:21.191016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc8ec7f-5e10-49f0-92a2-c28c7cd9c440 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:21.191296 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc8ec7f-5e10-49f0-92a2-c28c7cd9c440 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:21.210600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:21.225379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-bdc8ec7f-5e10-49f0-92a2-c28c7cd9c440 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:21.226602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bdc8ec7f-5e10-49f0-92a2-c28c7cd9c440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:21.228383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51581262-87bf-4c81-9969-e36f212186c9 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:21.270025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:21.270924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:21.271214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:21.271494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/117f0bab037044c6a6cb3ada31c08221/roles/bf3b74a9a9db444a8288ca37ccf736b7` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:21.276680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] RBAC: Authorizing `identity:create_grant(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user_id=117f0bab037044c6a6cb3ada31c08221, role_id=bf3b74a9a9db444a8288ca37ccf736b7)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:21.278743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:21.307469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 137/287] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:21 2022] PUT /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/users/117f0bab037044c6a6cb3ada31c08221/roles/bf3b74a9a9db444a8288ca37ccf736b7 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:21.342370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 151/288] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:21.381425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:21.525134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e05cc8-9773-48c4-beb5-8737f7f49fd0 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:21.567953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:21.568797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:21.569078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:21.569350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:21.584633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:21.592982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:21.593870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:21.595271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 138/289] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:21.600090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc8ec7f-5e10-49f0-92a2-c28c7cd9c440 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:21.729513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc8ec7f-5e10-49f0-92a2-c28c7cd9c440 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:21.759165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:21.773030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:21.773730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:21.774019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:21.774302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:21.775081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:21.775449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:21.776870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:21.793380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 152/290] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:21.797492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:21.812234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:21.914962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6cc0ee-6ffb-404f-a673-68031dd62f9e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:21.970451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:21.971331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:21.971615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:21.971891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] PATH_INFO: `/v3/users/6d19713fb78f4a2d9dcac5198356bc26` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:21.973124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] RBAC: Authorizing `identity:get_user(user_id=6d19713fb78f4a2d9dcac5198356bc26)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:21.974574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:21.976199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 139/291] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:21 2022] GET /identity/v3/users/6d19713fb78f4a2d9dcac5198356bc26 => generated 297 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:21.982247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:22.136667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4913171c-f2c3-4682-9ce1-17b4b2f1a0d6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:22.182119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:22.183157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:22.184144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:22.184648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:22.260133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:22.279893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:22.281711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:22.419685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 153/292] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:22.425501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:22.556766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:22.598898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27b5fad-c703-457e-a7e7-a321503acab7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:22.659850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bb4289-ac9c-4ff0-892f-1246c6052016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:22.660757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bb4289-ac9c-4ff0-892f-1246c6052016 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:22.661042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bb4289-ac9c-4ff0-892f-1246c6052016 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:22.661317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bb4289-ac9c-4ff0-892f-1246c6052016 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:22.662550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bb4289-ac9c-4ff0-892f-1246c6052016 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:22.664012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bb4289-ac9c-4ff0-892f-1246c6052016 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:22.665954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 140/293] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:22 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:22.672088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:22.783710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:22.802938 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c5f3d1-2f90-48c1-b19d-f4ac7fdfd4b7 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:22.854161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7781d53f-1cc3-4a54-bbdf-c817079249ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:22.854839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7781d53f-1cc3-4a54-bbdf-c817079249ef None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:22.855664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7781d53f-1cc3-4a54-bbdf-c817079249ef None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:22.855946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7781d53f-1cc3-4a54-bbdf-c817079249ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:22.856830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7781d53f-1cc3-4a54-bbdf-c817079249ef None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=6d19713fb78f4a2d9dcac5198356bc26) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:22.857146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7781d53f-1cc3-4a54-bbdf-c817079249ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:22.858602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7781d53f-1cc3-4a54-bbdf-c817079249ef None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:22.869323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 154/294] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:22 2022] GET /identity/v3/role_assignments?user.id=6d19713fb78f4a2d9dcac5198356bc26&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:23.008251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea043b-2669-4ed0-a778-864e6f86aa3e None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.008699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea043b-2669-4ed0-a778-864e6f86aa3e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.008869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdea043b-2669-4ed0-a778-864e6f86aa3e None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.010468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 141/295] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:23.017115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a52d266-f3b2-4a01-9f78-464d50928950 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.017687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a52d266-f3b2-4a01-9f78-464d50928950 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.018186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a52d266-f3b2-4a01-9f78-464d50928950 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.045371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5a52d266-f3b2-4a01-9f78-464d50928950 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:23.046904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5a52d266-f3b2-4a01-9f78-464d50928950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:23.061732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e3409-7b8a-49bc-a135-ba407484f119 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.062359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e3409-7b8a-49bc-a135-ba407484f119 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.062865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e3409-7b8a-49bc-a135-ba407484f119 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.068499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 142/296] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:23.078299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6d19b1-abf1-4046-9db7-8009f93f457b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.078907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6d19b1-abf1-4046-9db7-8009f93f457b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.079386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6d19b1-abf1-4046-9db7-8009f93f457b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.109097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5a6d19b1-abf1-4046-9db7-8009f93f457b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:23.110540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5a6d19b1-abf1-4046-9db7-8009f93f457b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:23.135036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 155/297] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:23.136554 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51578bfb-eabf-4df6-aba9-221b2e8cf650 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.136853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51578bfb-eabf-4df6-aba9-221b2e8cf650 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.137390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51578bfb-eabf-4df6-aba9-221b2e8cf650 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.138763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 156/298] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:23.145081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfefe71-fa43-4636-b5cc-904b1c27e209 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.145529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfefe71-fa43-4636-b5cc-904b1c27e209 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.146051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfefe71-fa43-4636-b5cc-904b1c27e209 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.171367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8dfefe71-fa43-4636-b5cc-904b1c27e209 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:23.171870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8dfefe71-fa43-4636-b5cc-904b1c27e209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:23.200259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 143/299] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:23.201421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6d19b1-abf1-4046-9db7-8009f93f457b None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.253829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 157/300] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:23.254705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfefe71-fa43-4636-b5cc-904b1c27e209 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.266773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:23.293561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:23.326277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6d19b1-abf1-4046-9db7-8009f93f457b None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.361085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.362327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.362816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.363159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.364409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:23.365116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfefe71-fa43-4636-b5cc-904b1c27e209 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.365868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:23.367749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 144/301] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:23.368570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.406242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.407188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.407441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.407715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.418264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:23.426510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:23.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:23.428169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 158/302] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:23.429320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.478523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b02ea28-684b-44c6-b1b9-6376b9ae4db3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.503720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:23.518572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.519499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.519789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.520075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.528162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:23.532103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:23.535792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a688a7a-2d30-429a-b5b9-ede7af281e10 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:23.540755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:23.541623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:23.542403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 145/303] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:23.543583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.576265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.576986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.577273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.577556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] PATH_INFO: `/v3/groups` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.578240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:23.579696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:23.593997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 159/304] 10.222.0.20 () {60 vars in 1134 bytes} [Tue Jul 5 09:57:23 2022] POST /identity/v3/groups => generated 229 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:23.594737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.666260 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52ffe24-2c70-4c8c-8198-efe0b9d9043e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.707771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.712433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.712433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.712433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.712433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] RBAC: Adding query filter params (name=member) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:23.712433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:23.712433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:23.731544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 146/305] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:23.735934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.763276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9d39af-a3f1-45d1-b2df-d336f3b2d537 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.803689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:23.814008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70b689a-dde8-4b2b-8602-89436a6b2970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.814766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b689a-dde8-4b2b-8602-89436a6b2970 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.815028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:23.815356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b689a-dde8-4b2b-8602-89436a6b2970 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.816500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b689a-dde8-4b2b-8602-89436a6b2970 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.817934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70b689a-dde8-4b2b-8602-89436a6b2970 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:23.818062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70b689a-dde8-4b2b-8602-89436a6b2970 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:23.820370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70b689a-dde8-4b2b-8602-89436a6b2970 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:23.830421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 160/306] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:23.831881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1b81b0-783c-4a63-9d63-4b672dd71e4e None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.832192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1b81b0-783c-4a63-9d63-4b672dd71e4e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.832510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1b81b0-783c-4a63-9d63-4b672dd71e4e None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.834121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 161/307] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:23.836991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1b81b0-783c-4a63-9d63-4b672dd71e4e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:23.865246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f837bb0f-3843-4fbd-90ef-b56ff41f9c85 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.903728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27c01a5-0335-4362-bca4-382c2f2c80c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.904646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c01a5-0335-4362-bca4-382c2f2c80c6 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.904927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c01a5-0335-4362-bca4-382c2f2c80c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.905199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c01a5-0335-4362-bca4-382c2f2c80c6 None admin] PATH_INFO: `/v3/users/11f055ec66ea43c7a70ce81ca351e99c` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.906402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27c01a5-0335-4362-bca4-382c2f2c80c6 None admin] RBAC: Authorizing `identity:get_user(user_id=11f055ec66ea43c7a70ce81ca351e99c)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:23.907848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27c01a5-0335-4362-bca4-382c2f2c80c6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:23.913849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 147/308] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:23 2022] GET /identity/v3/users/11f055ec66ea43c7a70ce81ca351e99c => generated 293 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:23.913849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67507c5d-426b-4be7-b91a-f6081d5ac840 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.913849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67507c5d-426b-4be7-b91a-f6081d5ac840 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.913849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67507c5d-426b-4be7-b91a-f6081d5ac840 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:23.930197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-67507c5d-426b-4be7-b91a-f6081d5ac840 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:23.931339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-67507c5d-426b-4be7-b91a-f6081d5ac840 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:23.955512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1b81b0-783c-4a63-9d63-4b672dd71e4e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:23.993079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:23.993880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:23.994168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:23.994450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] PATH_INFO: `/v3/users/a99441525c934c9fa12e1e3e852873e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:24.005074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 148/309] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:24.006078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67507c5d-426b-4be7-b91a-f6081d5ac840 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:24.013797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] RBAC: Authorizing `identity:get_user(user_id=a99441525c934c9fa12e1e3e852873e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:24.016269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:24.023402 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 162/310] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:23 2022] GET /identity/v3/users/a99441525c934c9fa12e1e3e852873e6 => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:24.025598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:24.075863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:24.077949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:24.124797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67507c5d-426b-4be7-b91a-f6081d5ac840 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:24.136478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e06c558-b5f9-4dc7-9530-4d03e2091f92 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:24.162167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:24.163036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:24.163318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:24.163598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] PATH_INFO: `/v3/system/users/11f055ec66ea43c7a70ce81ca351e99c/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:24.165389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=11f055ec66ea43c7a70ce81ca351e99c, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:24.166960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:24.172942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:24.173701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:24.173974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:24.174327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:24.175311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] RBAC: Adding query filter params (role.id=bf3b74a9a9db444a8288ca37ccf736b7, user.id=a99441525c934c9fa12e1e3e852873e6, scope.system=all) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:24.175693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:24.177259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:24.177926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 149/311] 10.222.0.20 () {58 vars in 1334 bytes} [Tue Jul 5 09:57:24 2022] PUT /identity/v3/system/users/11f055ec66ea43c7a70ce81ca351e99c/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:24.179046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:24.194297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 163/312] 10.222.0.20 () {56 vars in 1308 bytes} [Tue Jul 5 09:57:24 2022] GET /identity/v3/role_assignments?user.id=a99441525c934c9fa12e1e3e852873e6&scope.system=all&role.id=bf3b74a9a9db444a8288ca37ccf736b7 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:24.309871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26db1a8-6144-4631-8019-4ae3f1cdbe1a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:24.338584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:24.353554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:24.354383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:24.354668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:24.354973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:24.368020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:24.376456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:24.377402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:24.378453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 150/313] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:24.383589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:24.491028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874590e5-48af-41be-9f74-a0d69677571d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:24.507381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:24.529292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:24.530292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:24.530934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:24.531410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:24.532360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:24.532900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:24.534488 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:24.544382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 164/314] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:24.549450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:24.658566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a277ca9-59b4-4f29-9bf4-2558d438c426 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:24.693506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:24.694731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:24.695233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:24.695706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] PATH_INFO: `/v3/users/117f0bab037044c6a6cb3ada31c08221` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:24.697266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] RBAC: Authorizing `identity:get_user(user_id=117f0bab037044c6a6cb3ada31c08221)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:24.699064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:24.701338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 151/315] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:24 2022] GET /identity/v3/users/117f0bab037044c6a6cb3ada31c08221 => generated 297 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:24.707054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:24.813551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753eccc-d2e9-48ad-a1d8-9ae20063f860 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:24.848306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:24.849205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:24.849639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:24.850091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:24.896923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:24.922241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:24.923923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:25.026079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 165/316] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:25.032298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:25.086347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:25.137344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efad4932-61a0-4e79-864f-03b904078965 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:25.172453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5c7662-5314-4e27-97c3-cbab6abb770e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:25.173604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c7662-5314-4e27-97c3-cbab6abb770e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:25.174126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c7662-5314-4e27-97c3-cbab6abb770e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:25.174593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5c7662-5314-4e27-97c3-cbab6abb770e None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:25.175981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5c7662-5314-4e27-97c3-cbab6abb770e None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:25.177674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5c7662-5314-4e27-97c3-cbab6abb770e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:25.179585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 152/317] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:25 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:25.185667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:25.264519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:25.289092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ae4ba-5368-4a6e-ad8b-6eaaa28c399d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:25.324787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fe9df2-aef9-43e5-908b-bdada29f12fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:25.325451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fe9df2-aef9-43e5-908b-bdada29f12fa None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:25.325739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fe9df2-aef9-43e5-908b-bdada29f12fa None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:25.326018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fe9df2-aef9-43e5-908b-bdada29f12fa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:25.326858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fe9df2-aef9-43e5-908b-bdada29f12fa None admin] RBAC: Adding query filter params (role.id=bf3b74a9a9db444a8288ca37ccf736b7, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6, user.id=117f0bab037044c6a6cb3ada31c08221) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:25.327176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fe9df2-aef9-43e5-908b-bdada29f12fa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:25.328657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08fe9df2-aef9-43e5-908b-bdada29f12fa None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:25.335041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 166/318] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:25 2022] GET /identity/v3/role_assignments?user.id=117f0bab037044c6a6cb3ada31c08221&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=bf3b74a9a9db444a8288ca37ccf736b7 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:25.676601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f66018b-ef57-4c78-8c4c-db82079973a3 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:25.677106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f66018b-ef57-4c78-8c4c-db82079973a3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:25.677180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f66018b-ef57-4c78-8c4c-db82079973a3 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:25.678521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 153/319] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:25.684303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a993e7bd-46ba-4aa6-b8bd-1cc876d7f6d7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:25.684634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a993e7bd-46ba-4aa6-b8bd-1cc876d7f6d7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:25.684921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a993e7bd-46ba-4aa6-b8bd-1cc876d7f6d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:25.702067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a993e7bd-46ba-4aa6-b8bd-1cc876d7f6d7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:25.703231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a993e7bd-46ba-4aa6-b8bd-1cc876d7f6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:25.770319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 167/320] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:25.791931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f66018b-ef57-4c78-8c4c-db82079973a3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:25.898314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f66018b-ef57-4c78-8c4c-db82079973a3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:25.923337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:25.939917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daeb7b6-a287-4d64-a7d4-37df86a4f819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:25.940754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daeb7b6-a287-4d64-a7d4-37df86a4f819 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:25.941044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daeb7b6-a287-4d64-a7d4-37df86a4f819 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:25.941320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daeb7b6-a287-4d64-a7d4-37df86a4f819 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:25.942482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daeb7b6-a287-4d64-a7d4-37df86a4f819 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:25.943940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daeb7b6-a287-4d64-a7d4-37df86a4f819 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:25.945858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 154/321] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:25.951704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a993e7bd-46ba-4aa6-b8bd-1cc876d7f6d7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:26.023947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:26.057062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a993e7bd-46ba-4aa6-b8bd-1cc876d7f6d7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:26.091558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd2cb29-f368-4491-9481-2bfabdddbfe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:26.092233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd2cb29-f368-4491-9481-2bfabdddbfe3 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.092553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd2cb29-f368-4491-9481-2bfabdddbfe3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.092835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd2cb29-f368-4491-9481-2bfabdddbfe3 None admin] PATH_INFO: `/v3/groups` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.093528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd2cb29-f368-4491-9481-2bfabdddbfe3 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:26.094982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd2cb29-f368-4491-9481-2bfabdddbfe3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:26.116432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 168/322] 10.222.0.20 () {60 vars in 1134 bytes} [Tue Jul 5 09:57:25 2022] POST /identity/v3/groups => generated 226 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:26.316120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e795cf-0562-44a9-ac49-e06887fecb88 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.316458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e795cf-0562-44a9-ac49-e06887fecb88 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.316705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e795cf-0562-44a9-ac49-e06887fecb88 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.318253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 155/323] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:26.323221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5ec3c-f3dc-4f90-91a8-31004db17c0a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.323668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5ec3c-f3dc-4f90-91a8-31004db17c0a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.324372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5ec3c-f3dc-4f90-91a8-31004db17c0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.341258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-25a5ec3c-f3dc-4f90-91a8-31004db17c0a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:26.342459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-25a5ec3c-f3dc-4f90-91a8-31004db17c0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:26.404158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d786e0-0687-4582-bf0e-7b55c2cb4fe7 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.404547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d786e0-0687-4582-bf0e-7b55c2cb4fe7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.404834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d786e0-0687-4582-bf0e-7b55c2cb4fe7 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.406245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 156/324] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:57:26.412433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcd1e7-ea56-4083-a65b-4b6f7bd18599 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.412717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcd1e7-ea56-4083-a65b-4b6f7bd18599 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.413001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcd1e7-ea56-4083-a65b-4b6f7bd18599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.424192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 169/325] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:26.434507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-05fcd1e7-ea56-4083-a65b-4b6f7bd18599 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:26.435698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-05fcd1e7-ea56-4083-a65b-4b6f7bd18599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:26.450272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5ec3c-f3dc-4f90-91a8-31004db17c0a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:26.526057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:26.532647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 157/326] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:26.557493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fcd1e7-ea56-4083-a65b-4b6f7bd18599 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:26.583614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5ec3c-f3dc-4f90-91a8-31004db17c0a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:26.629299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:26.630809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.631096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.631388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.650282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:26.661591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:26.662529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:26.663313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:26.664382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 170/327] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:26.670002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:26.680082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fcd1e7-ea56-4083-a65b-4b6f7bd18599 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:26.702849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:26.724759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:26.725957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.726302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.726639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.740067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:26.749886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:26.751725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:26.753684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 158/328] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:26 2022] GET /identity/v3/roles/reader => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:26.760170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:26.794437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26b527d-6e31-49a7-975a-de015b407d20 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:26.816534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:26.833279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:26.834031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.834367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.834654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.837423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:26.837884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:26.839385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:26.852357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 171/329] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:26.860481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:26.882584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fbf6d0-e4f1-43bf-bfd2-175add6ade05 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:26.926790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:26.927476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:26.927758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:26.928029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:26.931273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] RBAC: Adding query filter params (name=reader) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:26.931273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:26.932677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:26.945227 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 159/330] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:26.956566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:26.999358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9bf523-5176-4099-b483-c74d77285066 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:27.004452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:27.037841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:27.038711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:27.038993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:27.039269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] PATH_INFO: `/v3/users/11f055ec66ea43c7a70ce81ca351e99c` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:27.040988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] RBAC: Authorizing `identity:get_user(user_id=11f055ec66ea43c7a70ce81ca351e99c)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:27.042433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:27.044118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 172/331] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:26 2022] GET /identity/v3/users/11f055ec66ea43c7a70ce81ca351e99c => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:27.050740 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:27.116856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd42aa-9c11-476e-b678-65d4bc09bdfd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:27.124851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:27.179282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a67e4-0ba3-42bd-b5d1-d20263c35f16 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:27.217039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:27.218167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:27.218596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:27.219017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] PATH_INFO: `/v3/users/a99441525c934c9fa12e1e3e852873e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:27.221110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] RBAC: Authorizing `identity:get_user(user_id=a99441525c934c9fa12e1e3e852873e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:27.223207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:27.227065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 160/332] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:26 2022] GET /identity/v3/users/a99441525c934c9fa12e1e3e852873e6 => generated 293 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:27.228724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d6579d-5c91-4e8c-b633-775d9742ca06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:27.229575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d6579d-5c91-4e8c-b633-775d9742ca06 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:27.230058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d6579d-5c91-4e8c-b633-775d9742ca06 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:27.230978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d6579d-5c91-4e8c-b633-775d9742ca06 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:27.231968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d6579d-5c91-4e8c-b633-775d9742ca06 None admin] RBAC: Adding query filter params (role.id=eefd36f11cba4b2eac18767b1b872d97, user.id=11f055ec66ea43c7a70ce81ca351e99c, scope.system=all) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:27.232899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:27.233712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d6579d-5c91-4e8c-b633-775d9742ca06 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:27.235634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d6579d-5c91-4e8c-b633-775d9742ca06 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:27.262859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 173/333] 10.222.0.20 () {56 vars in 1308 bytes} [Tue Jul 5 09:57:27 2022] GET /identity/v3/role_assignments?user.id=11f055ec66ea43c7a70ce81ca351e99c&scope.system=all&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 510 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:27.359435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb16462-b8ed-4cd2-b02e-22ad0875ed17 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:27.370415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:27.398949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aa724e-8f7f-4a95-b935-c7acc59b1ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:27.400113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa724e-8f7f-4a95-b935-c7acc59b1ffa None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:27.400578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa724e-8f7f-4a95-b935-c7acc59b1ffa None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:27.401004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa724e-8f7f-4a95-b935-c7acc59b1ffa None admin] PATH_INFO: `/v3/system/users/a99441525c934c9fa12e1e3e852873e6/roles/bf3b74a9a9db444a8288ca37ccf736b7` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:27.402735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aa724e-8f7f-4a95-b935-c7acc59b1ffa None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=a99441525c934c9fa12e1e3e852873e6, role_id=bf3b74a9a9db444a8288ca37ccf736b7)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:27.404291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aa724e-8f7f-4a95-b935-c7acc59b1ffa None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:27.419652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 161/334] 10.222.0.20 () {58 vars in 1334 bytes} [Tue Jul 5 09:57:27 2022] PUT /identity/v3/system/users/a99441525c934c9fa12e1e3e852873e6/roles/bf3b74a9a9db444a8288ca37ccf736b7 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:28.482617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee60845e-a93d-4412-922e-7f726577df92 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:28.483118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee60845e-a93d-4412-922e-7f726577df92 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:28.483380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee60845e-a93d-4412-922e-7f726577df92 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:28.484678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 174/335] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:28.490112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ee2e7a-821c-443d-b989-9a8a837d2a01 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:28.490512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ee2e7a-821c-443d-b989-9a8a837d2a01 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:28.490677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ee2e7a-821c-443d-b989-9a8a837d2a01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:28.509519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e6ee2e7a-821c-443d-b989-9a8a837d2a01 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:28.511057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e6ee2e7a-821c-443d-b989-9a8a837d2a01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:28.593065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 162/336] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:28.629113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee60845e-a93d-4412-922e-7f726577df92 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:28.744871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee60845e-a93d-4412-922e-7f726577df92 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:28.754865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:28.789839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:28.790658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:28.790945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:28.791280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:28.807020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:28.815235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:28.816104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:28.816939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 175/337] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:28.822211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ee2e7a-821c-443d-b989-9a8a837d2a01 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:28.959654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ee2e7a-821c-443d-b989-9a8a837d2a01 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:28.961225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:29.006482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1dd1f8-1a33-4422-a30c-c2244c04232a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:29.007083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1dd1f8-1a33-4422-a30c-c2244c04232a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.007366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1dd1f8-1a33-4422-a30c-c2244c04232a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.007643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1dd1f8-1a33-4422-a30c-c2244c04232a None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.008425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1dd1f8-1a33-4422-a30c-c2244c04232a None admin] RBAC: Adding query filter params (name=member) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:29.008730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1dd1f8-1a33-4422-a30c-c2244c04232a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:29.010125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1dd1f8-1a33-4422-a30c-c2244c04232a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:29.022526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 163/338] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:29.028253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:29.161573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23eefe9-5f44-4ae4-a508-986092f3c490 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:29.198811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:29.199476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.199765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.200046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.223224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118ad5ff-f3ab-4dcd-89a4-fbfc072b7e1d None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.223842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118ad5ff-f3ab-4dcd-89a4-fbfc072b7e1d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.224313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118ad5ff-f3ab-4dcd-89a4-fbfc072b7e1d None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.226122 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 164/339] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:29.231812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef639c56-d1ed-4294-b4bc-289ac8f99ffc None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.232348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef639c56-d1ed-4294-b4bc-289ac8f99ffc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.232840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef639c56-d1ed-4294-b4bc-289ac8f99ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.254268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:29.273487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ef639c56-d1ed-4294-b4bc-289ac8f99ffc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:29.274996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ef639c56-d1ed-4294-b4bc-289ac8f99ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:29.313478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:29.314976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:29.406453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 165/340] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:29.443220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef639c56-d1ed-4294-b4bc-289ac8f99ffc None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:29.499864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 176/341] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:29.506143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:29.611435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:29.616787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:29.622547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef639c56-d1ed-4294-b4bc-289ac8f99ffc None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:29.682993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:29.684378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.684689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.684965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.685771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba6c75b-d54c-4962-8f4d-a3f31abfe3b6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:29.703156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:29.716475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:29.717385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:29.718174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:29.720897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 166/342] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:29.726651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:29.777738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:29.778598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.779043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.779485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.780919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:29.782669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:29.784637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 177/343] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:29 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:29.801427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:29.842410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90433917-13b9-4115-a387-c0623db4c1c7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:29.886476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4e9dfe-facb-46bf-948a-b637dda14de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:29.887087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e9dfe-facb-46bf-948a-b637dda14de2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.887373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e9dfe-facb-46bf-948a-b637dda14de2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.887652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4e9dfe-facb-46bf-948a-b637dda14de2 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.888599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e9dfe-facb-46bf-948a-b637dda14de2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:29.888917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e9dfe-facb-46bf-948a-b637dda14de2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:29.890437 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4e9dfe-facb-46bf-948a-b637dda14de2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:29.914965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 167/344] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:29.916591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c023f18-9405-4f92-8ac6-bb50fd5aa8a5 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:29.916964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c023f18-9405-4f92-8ac6-bb50fd5aa8a5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:29.917250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c023f18-9405-4f92-8ac6-bb50fd5aa8a5 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:29.928151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 168/345] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:29.928151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c023f18-9405-4f92-8ac6-bb50fd5aa8a5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:29.959191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:29.967164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567427d2-0f95-4a5b-afaf-05871465ed07 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:30.009977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fc77b3-0b2c-49a3-8b36-982a461341f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:30.012164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fc77b3-0b2c-49a3-8b36-982a461341f0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.014847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fc77b3-0b2c-49a3-8b36-982a461341f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.015490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fc77b3-0b2c-49a3-8b36-982a461341f0 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.020370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:30.024116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fc77b3-0b2c-49a3-8b36-982a461341f0 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:30.025972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46fc77b3-0b2c-49a3-8b36-982a461341f0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:30.031227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 178/346] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:29 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:30.032745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274f4a45-4360-4d70-a0a7-b7482ac3b917 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.033198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274f4a45-4360-4d70-a0a7-b7482ac3b917 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.034232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274f4a45-4360-4d70-a0a7-b7482ac3b917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.055592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-274f4a45-4360-4d70-a0a7-b7482ac3b917 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:30.057069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-274f4a45-4360-4d70-a0a7-b7482ac3b917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:30.139320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 179/347] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:30.140455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274f4a45-4360-4d70-a0a7-b7482ac3b917 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:30.192828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c023f18-9405-4f92-8ac6-bb50fd5aa8a5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:30.273751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:30.274435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.274719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.275003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.277668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274f4a45-4360-4d70-a0a7-b7482ac3b917 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:30.310903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:30.317513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:30.318332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.318719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.319005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.319854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=68b3ea181434496b91ea5053896483e8) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:30.321213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:30.322844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:30.329765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 180/348] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:30 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 260 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:30.330602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:30.388779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:30.390367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:30.413301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:30.453379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c879ce6-a6a8-4788-8bd1-22f46352f8fa None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:30.491792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:30.492873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.493160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.500589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.513999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:30.522753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:30.523952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:30.524863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:30.525563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 181/349] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:30.531663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:30.565019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 169/350] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 640 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:30.570036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:30.595193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:30.646508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd47dc9-a5b1-4ece-a86d-f8068b72de0c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:30.683525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc5ac2b-f311-4714-828d-b3ab0df68d86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:30.684207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc5ac2b-f311-4714-828d-b3ab0df68d86 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.684531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc5ac2b-f311-4714-828d-b3ab0df68d86 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.684794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc5ac2b-f311-4714-828d-b3ab0df68d86 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.685552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc5ac2b-f311-4714-828d-b3ab0df68d86 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:30.685861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc5ac2b-f311-4714-828d-b3ab0df68d86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:30.687239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc5ac2b-f311-4714-828d-b3ab0df68d86 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:30.696725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 182/351] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:30.697837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a80c9d1-d78c-4b72-8e76-12429ea3203d None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.698125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a80c9d1-d78c-4b72-8e76-12429ea3203d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.698411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a80c9d1-d78c-4b72-8e76-12429ea3203d None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.699710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 183/352] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:30.705953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb598841-fb56-4391-88dc-41dcb742c3ed None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.706252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb598841-fb56-4391-88dc-41dcb742c3ed None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.706491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb598841-fb56-4391-88dc-41dcb742c3ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.724559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-bb598841-fb56-4391-88dc-41dcb742c3ed None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:30.725201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79739622-6a9a-4fb6-b00b-7ebcb68b2a2c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:30.725754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bb598841-fb56-4391-88dc-41dcb742c3ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:30.752588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:30.777846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:30.779185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:30.779292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:30.779550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:30.782497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:30.784368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:30.786923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 170/353] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:30 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:30.787788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:30.804997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 184/354] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:30.806231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb598841-fb56-4391-88dc-41dcb742c3ed None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:30.880805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:30.972532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe5bce5-9261-4f9a-bd77-ec041b98c7a6 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:30.973568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb598841-fb56-4391-88dc-41dcb742c3ed None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.022338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed11f32d-e1a1-4f13-addf-c3e0b56308b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.023520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11f32d-e1a1-4f13-addf-c3e0b56308b8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.023956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11f32d-e1a1-4f13-addf-c3e0b56308b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.024881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed11f32d-e1a1-4f13-addf-c3e0b56308b8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.030366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.031392 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.031754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.032106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.033482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:31.034999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:31.036784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 185/355] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:30 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:31.037872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:31.102400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:31.126847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11f32d-e1a1-4f13-addf-c3e0b56308b8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:31.128338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed11f32d-e1a1-4f13-addf-c3e0b56308b8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:31.182475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0b05b2-dd7a-418b-8cce-2b880e260be2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.204721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:31.243044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.243861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.244148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.244457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.258022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:31.263566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 171/356] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:31.265248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a87e5a2-3c8c-4eef-94db-732e58fed586 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.265765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a87e5a2-3c8c-4eef-94db-732e58fed586 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.266505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a87e5a2-3c8c-4eef-94db-732e58fed586 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.269045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 172/357] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:31.269326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:31.270182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:31.270964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 186/358] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:31.271624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:31.272759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a87e5a2-3c8c-4eef-94db-732e58fed586 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:31.383128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:31.407931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291627a7-be61-4509-aa5e-2863dae64bff None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.410357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a87e5a2-3c8c-4eef-94db-732e58fed586 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.451816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.452630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.452877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457d96bb-e254-4b99-8259-88e7f6dff403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.452992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.453337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.454293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457d96bb-e254-4b99-8259-88e7f6dff403 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.454409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=68b3ea181434496b91ea5053896483e8) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:31.454692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:31.455088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457d96bb-e254-4b99-8259-88e7f6dff403 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.455550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457d96bb-e254-4b99-8259-88e7f6dff403 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.456417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:31.457135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457d96bb-e254-4b99-8259-88e7f6dff403 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:31.458993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457d96bb-e254-4b99-8259-88e7f6dff403 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:31.461766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 187/359] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:31 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:31.463203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e7f887-cc1b-4cd5-9bab-f81fc89d132e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.463674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e7f887-cc1b-4cd5-9bab-f81fc89d132e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.464050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 173/360] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:31 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 260 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:31.464319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e7f887-cc1b-4cd5-9bab-f81fc89d132e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.465095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:31.485550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-58e7f887-cc1b-4cd5-9bab-f81fc89d132e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:31.487157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-58e7f887-cc1b-4cd5-9bab-f81fc89d132e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:31.532671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:31.570248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 188/361] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:31.577479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e7f887-cc1b-4cd5-9bab-f81fc89d132e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:31.632719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aa2990-6b30-4bb9-b0c0-78eda09937d9 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.688311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5691da2-dcfd-4284-84db-0d36e6c89d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.689028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5691da2-dcfd-4284-84db-0d36e6c89d95 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.689315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5691da2-dcfd-4284-84db-0d36e6c89d95 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.689750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5691da2-dcfd-4284-84db-0d36e6c89d95 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.690520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5691da2-dcfd-4284-84db-0d36e6c89d95 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:31.691159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5691da2-dcfd-4284-84db-0d36e6c89d95 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:31.693051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5691da2-dcfd-4284-84db-0d36e6c89d95 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:31.696908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e7f887-cc1b-4cd5-9bab-f81fc89d132e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.703372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 174/362] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:31.704787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17115eb-4719-4763-a696-394349a91f2c None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.705312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17115eb-4719-4763-a696-394349a91f2c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.705546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17115eb-4719-4763-a696-394349a91f2c None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.707051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 175/363] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:31.707880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17115eb-4719-4763-a696-394349a91f2c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:31.728284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:31.745374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.746504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.747043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.747467 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.750114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:31.751884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:31.753738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 189/364] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:31 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:31.754767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:31.809043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:31.849820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17115eb-4719-4763-a696-394349a91f2c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.878858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd6ffa3-476f-4646-b418-353ae9c4625f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:31.900833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cd2368-397a-49f3-8658-9b9c047a7eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.901930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd2368-397a-49f3-8658-9b9c047a7eee None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.902749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd2368-397a-49f3-8658-9b9c047a7eee None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.903031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cd2368-397a-49f3-8658-9b9c047a7eee None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.917700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cd2368-397a-49f3-8658-9b9c047a7eee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-05cd2368-397a-49f3-8658-9b9c047a7eee None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:31.925899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:31.926826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:31.927628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 176/365] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:31.928470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022aafd-eef7-4c54-98fd-9b843d2e3aa5 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.932414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022aafd-eef7-4c54-98fd-9b843d2e3aa5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.932414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4022aafd-eef7-4c54-98fd-9b843d2e3aa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.933237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:31.934456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:31.934858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:31.935355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:31.948078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4022aafd-eef7-4c54-98fd-9b843d2e3aa5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:31.950395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4022aafd-eef7-4c54-98fd-9b843d2e3aa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:32.020307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:32.022088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:32.037492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:32.044852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 177/366] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:32.048142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4022aafd-eef7-4c54-98fd-9b843d2e3aa5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.080326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:32.155303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 190/367] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:32.156218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.184725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4022aafd-eef7-4c54-98fd-9b843d2e3aa5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:32.224186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:32.224910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:32.225193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:32.225473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:32.226377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:32.226689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:32.228201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:32.234856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 178/368] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 260 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:32.235652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.282461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad18ec11-50bb-4808-bc7d-0a5cb2a18997 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:32.325119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:32.325797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:32.326086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:32.326363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:32.327578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] RBAC: Adding query filter params (name=reader) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:32.327901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:32.329365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:32.339752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 191/369] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:32.340739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.344578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:32.378047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:32.445619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11acb003-6faa-4ae1-8924-e0f61073ae92 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:32.470891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3710b647-6ea9-42c4-aa1b-21015988122d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:32.489556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:32.490358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:32.490637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:32.490917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:32.502320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:32.510564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:32.513098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 179/370] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:32.514268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.522058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:32.522710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:32.522996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:32.523274 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:32.524574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:32.526053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:32.528379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 192/371] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:32.530439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.633078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:32.656730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1b0b2-4388-43e4-a1d2-41aeced83cc4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:32.677386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd60740-0b19-45bb-9563-0fd157cfcad7 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:32.684257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:32.714209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:32.715745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:32.716028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:32.716302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] PATH_INFO: `/v3/users/a99441525c934c9fa12e1e3e852873e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:32.717536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] RBAC: Authorizing `identity:get_user(user_id=a99441525c934c9fa12e1e3e852873e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:32.719000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:32.720704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 180/372] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/users/a99441525c934c9fa12e1e3e852873e6 => generated 293 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:32.721564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.744262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:32.745209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:32.746276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:32.746846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:32.747803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:32.748305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:32.750041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:32.768493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 193/373] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:32.769811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.831771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c0d55c-96a9-4214-861e-e49b3b23827b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:32.870453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:32.872318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:32.872840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:32.873315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:32.874735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:32.876370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:32.878601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 181/374] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:32.879615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:32.963641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb8dcd2-575e-4b2a-88c8-b0d93ad3fe5d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:33.001052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:33.021124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:33.033097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:33.033773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.034063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.034349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.035321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] RBAC: Adding query filter params (role.id=bf3b74a9a9db444a8288ca37ccf736b7, user.id=a99441525c934c9fa12e1e3e852873e6, scope.system=all) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:33.035647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:33.037299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:33.056112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d84b9da-78f9-4f37-af1d-3e74437550f9 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:33.060040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 194/375] 10.222.0.20 () {56 vars in 1308 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/role_assignments?user.id=a99441525c934c9fa12e1e3e852873e6&scope.system=all&role.id=bf3b74a9a9db444a8288ca37ccf736b7 => generated 510 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:33.065328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:33.154110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725ac6fb-5005-42cd-a077-1b8c3b70400d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:33.154786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ac6fb-5005-42cd-a077-1b8c3b70400d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.155074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ac6fb-5005-42cd-a077-1b8c3b70400d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.155353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725ac6fb-5005-42cd-a077-1b8c3b70400d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.241811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725ac6fb-5005-42cd-a077-1b8c3b70400d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:33.243975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725ac6fb-5005-42cd-a077-1b8c3b70400d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:33.322351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:33.347829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409ec697-592c-4c4d-a047-c7d02da04c2a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:33.363663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 182/376] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:33.364853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43edc02c-9654-4566-b938-63fdb0c0b5e7 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.365211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43edc02c-9654-4566-b938-63fdb0c0b5e7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.365555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43edc02c-9654-4566-b938-63fdb0c0b5e7 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.367971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 183/377] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:33.373285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43edc02c-9654-4566-b938-63fdb0c0b5e7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:33.400464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d6d45e-a923-44c7-911d-be91c18d23ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:33.401478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6d45e-a923-44c7-911d-be91c18d23ac None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.402026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6d45e-a923-44c7-911d-be91c18d23ac None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.402975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d6d45e-a923-44c7-911d-be91c18d23ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.403996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d6d45e-a923-44c7-911d-be91c18d23ac None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:33.404565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d6d45e-a923-44c7-911d-be91c18d23ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:33.406176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d6d45e-a923-44c7-911d-be91c18d23ac None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:33.414983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 195/378] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:33 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 260 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:33.416759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173269d-9202-4b46-9828-b3a13d5273b5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.417236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173269d-9202-4b46-9828-b3a13d5273b5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.417728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5173269d-9202-4b46-9828-b3a13d5273b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.443644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5173269d-9202-4b46-9828-b3a13d5273b5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:33.445790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5173269d-9202-4b46-9828-b3a13d5273b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:33.515876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43edc02c-9654-4566-b938-63fdb0c0b5e7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:33.525989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:33.557218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 196/379] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:33.570240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:33.571516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.572038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.572564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] PATH_INFO: `/v3/projects/ba1e700aae1b46039e859b8f609bb931` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.574664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] RBAC: Authorizing `identity:get_project(project_id=ba1e700aae1b46039e859b8f609bb931)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:33.576480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:33.578673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 184/380] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:33 2022] GET /identity/v3/projects/ba1e700aae1b46039e859b8f609bb931 => generated 344 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:33.583970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5173269d-9202-4b46-9828-b3a13d5273b5 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:33.584541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:33.636712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:33.761081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5173269d-9202-4b46-9828-b3a13d5273b5 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:33.776528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd27475-43ab-44ec-8e39-8629f7452c6a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:33.824182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:33.825004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.825283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.825551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.838888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:33.840306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:33.840608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:33.840883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] PATH_INFO: `/v3/groups/772e5a0905bf4f68ba80c5614360ee22` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:33.845646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:33.853397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:33.854279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:33.855651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:33.861427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 197/381] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:33.865154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] RBAC: Authorizing `identity:get_group(group_id=772e5a0905bf4f68ba80c5614360ee22)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:33.866618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:33.868340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 185/382] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:33 2022] GET /identity/v3/groups/772e5a0905bf4f68ba80c5614360ee22 => generated 229 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:33.868753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:33.877351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:33.936747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:33.999316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b7aa0f-0893-4fbc-bbda-056d28871df3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.016531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:34.031754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c817c3-7b47-4c30-8461-7a9d13aa8170 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.049964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6bc64b-6b04-4887-9917-36b95d00cd28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.051030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6bc64b-6b04-4887-9917-36b95d00cd28 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.051318 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6bc64b-6b04-4887-9917-36b95d00cd28 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.051597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6bc64b-6b04-4887-9917-36b95d00cd28 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.052362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6bc64b-6b04-4887-9917-36b95d00cd28 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:34.052695 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6bc64b-6b04-4887-9917-36b95d00cd28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:34.054762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6bc64b-6b04-4887-9917-36b95d00cd28 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:34.064814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 198/383] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:34.066216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0260b509-9387-42a4-bb7c-98089647b14f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.068786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0260b509-9387-42a4-bb7c-98089647b14f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.069075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0260b509-9387-42a4-bb7c-98089647b14f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.070661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 199/384] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:34.077195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0260b509-9387-42a4-bb7c-98089647b14f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.079942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52beb1c6-b026-4f87-8825-7f823a650f30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.080738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52beb1c6-b026-4f87-8825-7f823a650f30 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.081087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52beb1c6-b026-4f87-8825-7f823a650f30 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.081430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52beb1c6-b026-4f87-8825-7f823a650f30 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.082348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52beb1c6-b026-4f87-8825-7f823a650f30 None admin] RBAC: Adding query filter params (group.id=772e5a0905bf4f68ba80c5614360ee22, role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=ba1e700aae1b46039e859b8f609bb931) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:34.082725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52beb1c6-b026-4f87-8825-7f823a650f30 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:34.084231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52beb1c6-b026-4f87-8825-7f823a650f30 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:34.094308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 186/385] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:33 2022] GET /identity/v3/role_assignments?group.id=772e5a0905bf4f68ba80c5614360ee22&scope.project.id=ba1e700aae1b46039e859b8f609bb931&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 260 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:34.096782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29db9d-e1d5-4ccd-936c-a2ffb9b7d1cf None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.097388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29db9d-e1d5-4ccd-936c-a2ffb9b7d1cf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.097863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29db9d-e1d5-4ccd-936c-a2ffb9b7d1cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.122564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ce29db9d-e1d5-4ccd-936c-a2ffb9b7d1cf None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:34.124528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ce29db9d-e1d5-4ccd-936c-a2ffb9b7d1cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:34.214635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 187/386] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:34.242219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce29db9d-e1d5-4ccd-936c-a2ffb9b7d1cf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.254660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0260b509-9387-42a4-bb7c-98089647b14f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.268800 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:34.328848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.329416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.329701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.329976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.331166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:34.332685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:34.335018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 200/387] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:34 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:34.340767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.347424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:34.365357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce29db9d-e1d5-4ccd-936c-a2ffb9b7d1cf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.406902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.408246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.408770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.409259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.422147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:34.431985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:34.432979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:34.433842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:34.437239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 188/388] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:34.443713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.561076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55616e8-7820-4a43-87af-4f66a1ce85f1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.571333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62a21b6-19ba-4eea-be0f-3c50f89f7155 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.601087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.602133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.602628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.603100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.604081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:34.604613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:34.606475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:34.621507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 189/389] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:34.623456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.636200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:34.665505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.666374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.666654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.666928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.668179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:34.669661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:34.671437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 201/390] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:34 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:34.681516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.744684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d891b014-65d0-464d-807b-fe77cb1522b4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.764768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:34.786083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.787324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.787821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.788298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.790120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:34.791889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:34.794760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 190/391] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:34 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:34.800287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.890379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:34.906156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652c582b-b914-4f0f-9d67-aca36ef1860f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.919524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79df4fad-6f15-4307-9288-7c1f4c17550f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:34.958831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.959798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.960157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.960532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:34.962029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:34.963611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:34.966902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 191/392] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:34 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:34.972682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:34.997277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05b7fc0-8a1e-4d98-bb25-ccf19f603b0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:34.998167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05b7fc0-8a1e-4d98-bb25-ccf19f603b0e None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:34.998450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05b7fc0-8a1e-4d98-bb25-ccf19f603b0e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:34.998722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05b7fc0-8a1e-4d98-bb25-ccf19f603b0e None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:35.000789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05b7fc0-8a1e-4d98-bb25-ccf19f603b0e None admin] RBAC: Authorizing `identity:create_grant(project_id=68b3ea181434496b91ea5053896483e8, group_id=e0b51cfe57224b8eb97896476baa3bc4, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:35.002217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05b7fc0-8a1e-4d98-bb25-ccf19f603b0e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:35.035457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 202/393] 10.222.0.20 () {58 vars in 1442 bytes} [Tue Jul 5 09:57:34 2022] PUT /identity/v3/projects/68b3ea181434496b91ea5053896483e8/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:35.036803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d8b2d7-dfc9-41a1-86c4-f338f55a6375 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:35.037093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d8b2d7-dfc9-41a1-86c4-f338f55a6375 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:35.037376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d8b2d7-dfc9-41a1-86c4-f338f55a6375 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:35.038892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 203/394] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:35.046673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1da7566-322f-4f9c-bc0a-d7b4776f14f7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:35.046965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1da7566-322f-4f9c-bc0a-d7b4776f14f7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:35.047232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1da7566-322f-4f9c-bc0a-d7b4776f14f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:35.084381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a1da7566-322f-4f9c-bc0a-d7b4776f14f7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:35.085603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a1da7566-322f-4f9c-bc0a-d7b4776f14f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:35.091256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726aa274-6363-46b0-aa33-d7ffada0f92f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:35.133121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:35.134307 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:35.134595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:35.134880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:35.138586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] RBAC: Authorizing `identity:create_grant(project_id=68b3ea181434496b91ea5053896483e8, group_id=e0b51cfe57224b8eb97896476baa3bc4, role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:35.140175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:35.160376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 192/395] 10.222.0.20 () {58 vars in 1442 bytes} [Tue Jul 5 09:57:34 2022] PUT /identity/v3/projects/68b3ea181434496b91ea5053896483e8/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 09:57:35.247324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 204/396] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:35.283553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:35.287881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:35.406173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0426a7c3-44fb-49be-ae84-441498d8641c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:35.442757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:35.443801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:35.444243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:35.444689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] PATH_INFO: `/v3/roles/member` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:35.455531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:35.463539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:35.467862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:35.468868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:35.471013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 193/397] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:35.477008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1da7566-322f-4f9c-bc0a-d7b4776f14f7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:35.514212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:35.596646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1da7566-322f-4f9c-bc0a-d7b4776f14f7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:35.638369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:35.639040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:35.639323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:35.639598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:35.640344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:35.640673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:35.642305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:35.654332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 205/398] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:35.659687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:35.736138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:35.788103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a669b9b0-6d5e-44d4-8d84-5c1fcb08967d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:35.825627 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:35.826509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:35.826792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:35.827068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:35.828256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:35.829842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:35.831870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 194/399] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:35 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:35.837066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:35.969450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:35.977367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491baf8-8fa9-4e87-bcda-2cee71999edc None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:36.022281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3679ada4-8464-4496-acca-bd8a4ef7b743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:36.023076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679ada4-8464-4496-acca-bd8a4ef7b743 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.023364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679ada4-8464-4496-acca-bd8a4ef7b743 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.023640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3679ada4-8464-4496-acca-bd8a4ef7b743 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.028223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3679ada4-8464-4496-acca-bd8a4ef7b743 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:36.029777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3679ada4-8464-4496-acca-bd8a4ef7b743 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:36.031925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 206/400] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:35 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:36.037623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:36.169012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964e1ac2-0df7-4e2a-bc13-1fef7c3cd25a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:36.200761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:36.208155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21069b7d-a3dc-4464-8bc4-cadc0814defb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:36.209382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21069b7d-a3dc-4464-8bc4-cadc0814defb None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.209868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21069b7d-a3dc-4464-8bc4-cadc0814defb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.210328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21069b7d-a3dc-4464-8bc4-cadc0814defb None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.212539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21069b7d-a3dc-4464-8bc4-cadc0814defb None admin] RBAC: Authorizing `identity:create_grant(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6, group_id=e0b51cfe57224b8eb97896476baa3bc4, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:36.214196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21069b7d-a3dc-4464-8bc4-cadc0814defb None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:36.232135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 195/401] 10.222.0.20 () {58 vars in 1442 bytes} [Tue Jul 5 09:57:36 2022] PUT /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:36.446246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db652a4-aa1b-4417-ac70-d54fd9b2ce4f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.446535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db652a4-aa1b-4417-ac70-d54fd9b2ce4f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.446820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db652a4-aa1b-4417-ac70-d54fd9b2ce4f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.448212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 207/402] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:36.455067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbc454-749e-4c11-9ece-a7fc990dbf38 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.455638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbc454-749e-4c11-9ece-a7fc990dbf38 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.456131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cbc454-749e-4c11-9ece-a7fc990dbf38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.477875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b6cbc454-749e-4c11-9ece-a7fc990dbf38 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:36.479074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b6cbc454-749e-4c11-9ece-a7fc990dbf38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:36.481581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5b4b9d-6a34-4f90-b218-d42c98daad48 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.481932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5b4b9d-6a34-4f90-b218-d42c98daad48 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.482219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5b4b9d-6a34-4f90-b218-d42c98daad48 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.483839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 208/403] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:36.495699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbabc91-458c-4a15-9dde-b1d2fd7c12b1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.495985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbabc91-458c-4a15-9dde-b1d2fd7c12b1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.496270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbabc91-458c-4a15-9dde-b1d2fd7c12b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.518251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-bdbabc91-458c-4a15-9dde-b1d2fd7c12b1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:36.519450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bdbabc91-458c-4a15-9dde-b1d2fd7c12b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:36.575643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 196/404] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:36.609183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cbc454-749e-4c11-9ece-a7fc990dbf38 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:36.627848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 209/405] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:36.675568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbabc91-458c-4a15-9dde-b1d2fd7c12b1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:36.767572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cbc454-749e-4c11-9ece-a7fc990dbf38 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:36.810519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:36.828767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:36.831541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:36.832371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.832683 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.832964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.847066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbabc91-458c-4a15-9dde-b1d2fd7c12b1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:36.858223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:36.866489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:36.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:36.868180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 197/406] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:36.873767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:36.917580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:36.919264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:36.919771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:36.920248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:36.945710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:36.953745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:36.954642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:36.955866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 210/407] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:36.964250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:37.001075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f790c6-72cf-4087-a84d-1e467b182cfd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:37.072276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:37.072907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.073163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.073439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.075461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:37.075985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:37.077411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:37.093576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 198/408] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:37.098895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:37.186310 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d3d01-68d3-4558-a7bb-8e3cd8abab18 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:37.200251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80379bc-b1d0-42b9-bc6e-1edda3da801a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:37.216803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:37.235511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:37.236371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.236701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.236985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.238407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:37.240152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:37.243242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 199/409] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:37 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:37.249658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:37.337203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:37.337812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.338094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.338369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.339131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:37.340038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:37.341457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:37.357544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012946bd-8043-4657-859e-e3c940fb2509 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:37.366607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 211/410] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 403 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:37.371535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:37.373325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:37.396065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:37.396967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.397769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.398060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.399330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:37.400879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:37.403724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 200/411] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:37 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:37.409220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:37.425898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:37.525656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe90de3-94b3-424a-be65-a93e33caecc3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:37.563784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:37.564722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.565008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.565285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.567638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] RBAC: Authorizing `identity:create_grant(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6, group_id=e0b51cfe57224b8eb97896476baa3bc4, role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:37.569301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:37.584872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 201/412] 10.222.0.20 () {58 vars in 1442 bytes} [Tue Jul 5 09:57:37 2022] PUT /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6/groups/e0b51cfe57224b8eb97896476baa3bc4/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 09:57:37.648891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09918cf9-8fed-40f5-9c89-fff93bee8789 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:37.703690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:37.704619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.704910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.705193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] PATH_INFO: `/v3/groups/772e5a0905bf4f68ba80c5614360ee22` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.706418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] RBAC: Authorizing `identity:get_group(group_id=772e5a0905bf4f68ba80c5614360ee22)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:37.707963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:37.710042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 212/413] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:37 2022] GET /identity/v3/groups/772e5a0905bf4f68ba80c5614360ee22 => generated 229 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:37.716194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:37.782533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:37.845321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b833b5-9fe0-47ba-8942-a6ea6fd80794 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:37.880955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22b0845-addc-41ea-b691-49b5bab85166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:37.882280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b0845-addc-41ea-b691-49b5bab85166 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.882567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b0845-addc-41ea-b691-49b5bab85166 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.882846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22b0845-addc-41ea-b691-49b5bab85166 None admin] PATH_INFO: `/v3/projects/ba1e700aae1b46039e859b8f609bb931` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.884055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b0845-addc-41ea-b691-49b5bab85166 None admin] RBAC: Authorizing `identity:get_project(project_id=ba1e700aae1b46039e859b8f609bb931)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:37.885551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22b0845-addc-41ea-b691-49b5bab85166 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:37.888093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 202/414] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:37 2022] GET /identity/v3/projects/ba1e700aae1b46039e859b8f609bb931 => generated 344 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:37.895440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:37.963531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0b5342-f009-4467-8b6b-f57945058853 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.964115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0b5342-f009-4467-8b6b-f57945058853 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.965167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0b5342-f009-4467-8b6b-f57945058853 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.967927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 203/415] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:37.972158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892d1bc-0bcf-4565-aedb-6a7533bd16d5 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:37.972666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892d1bc-0bcf-4565-aedb-6a7533bd16d5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:37.973155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892d1bc-0bcf-4565-aedb-6a7533bd16d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:37.999903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-3892d1bc-0bcf-4565-aedb-6a7533bd16d5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:38.001126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3892d1bc-0bcf-4565-aedb-6a7533bd16d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:38.055629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:38.092684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b7c82b-e0c7-4c3d-a70b-4ade471a41e4 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:38.150399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb162e4-8e7f-48ad-88f7-d2b19700fd44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:38.151311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb162e4-8e7f-48ad-88f7-d2b19700fd44 None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.151596 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb162e4-8e7f-48ad-88f7-d2b19700fd44 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.152419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb162e4-8e7f-48ad-88f7-d2b19700fd44 None admin] PATH_INFO: `/v3/projects/ba1e700aae1b46039e859b8f609bb931/groups/772e5a0905bf4f68ba80c5614360ee22/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.152829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 204/416] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:38.154580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb162e4-8e7f-48ad-88f7-d2b19700fd44 None admin] RBAC: Authorizing `identity:create_grant(project_id=ba1e700aae1b46039e859b8f609bb931, group_id=772e5a0905bf4f68ba80c5614360ee22, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:38.156246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb162e4-8e7f-48ad-88f7-d2b19700fd44 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:38.178286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3892d1bc-0bcf-4565-aedb-6a7533bd16d5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:38.179476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 213/417] 10.222.0.20 () {58 vars in 1442 bytes} [Tue Jul 5 09:57:37 2022] PUT /identity/v3/projects/ba1e700aae1b46039e859b8f609bb931/groups/772e5a0905bf4f68ba80c5614360ee22/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:38.189041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fab6e80-c075-47d2-b445-ad1128fe36ec None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.189689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fab6e80-c075-47d2-b445-ad1128fe36ec None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.190187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fab6e80-c075-47d2-b445-ad1128fe36ec None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.192613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 214/418] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:57:38.199125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45380e-b86a-4c51-b395-4dc1d4ec2470 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.199915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45380e-b86a-4c51-b395-4dc1d4ec2470 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.200426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45380e-b86a-4c51-b395-4dc1d4ec2470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.228011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5d45380e-b86a-4c51-b395-4dc1d4ec2470 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:38.230738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5d45380e-b86a-4c51-b395-4dc1d4ec2470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:38.258249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:38.322194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 215/419] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:38.350145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d45380e-b86a-4c51-b395-4dc1d4ec2470 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:38.399634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3892d1bc-0bcf-4565-aedb-6a7533bd16d5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:38.435522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:38.436666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:38.437802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.438350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.438831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.452045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:38.460682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:38.461584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:38.462482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:38.465899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 205/420] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:38.471731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:38.471873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d45380e-b86a-4c51-b395-4dc1d4ec2470 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:38.509076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:38.510020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.510434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.510816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.522060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:38.530024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:38.531144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:38.531939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:38.532575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 216/421] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:38.538253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:38.621317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:38.624806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb009ddc-99db-4dec-ac3e-197f6582e3dd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:38.645594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4ff3c3-b142-4631-a912-7da75d849cf3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:38.676873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:38.681815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:38.682505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.682801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.683090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.683915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] RBAC: Adding query filter params (name=member) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:38.684381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:38.686120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:38.695738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 217/422] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:38.701782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:38.717129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:38.717830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.718111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.718386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.719153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:38.719459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:38.720875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:38.732776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 206/423] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:38.737978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:38.806268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62a350b-0874-4d14-8aea-455d0bbb3983 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:38.848908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638122d-f43d-449f-85a2-9d2e8fda76dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:38.850294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638122d-f43d-449f-85a2-9d2e8fda76dc None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.850738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638122d-f43d-449f-85a2-9d2e8fda76dc None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.851169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638122d-f43d-449f-85a2-9d2e8fda76dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.886196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2499415f-aa96-41ba-b593-1bf8820810aa None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:38.929828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:38.938423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b294d82-b7ad-45f2-b8d9-2980e83e33f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:38.939024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b294d82-b7ad-45f2-b8d9-2980e83e33f9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:38.939308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b294d82-b7ad-45f2-b8d9-2980e83e33f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:38.939588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b294d82-b7ad-45f2-b8d9-2980e83e33f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:38.942373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638122d-f43d-449f-85a2-9d2e8fda76dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:38.943852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638122d-f43d-449f-85a2-9d2e8fda76dc None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:38.983067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:39.045222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b294d82-b7ad-45f2-b8d9-2980e83e33f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:39.047224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b294d82-b7ad-45f2-b8d9-2980e83e33f9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:39.058496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 218/424] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:39.059725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a38a38-5c71-4ddd-8d3c-e127c65ff72e None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.060017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a38a38-5c71-4ddd-8d3c-e127c65ff72e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.060300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a38a38-5c71-4ddd-8d3c-e127c65ff72e None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.061749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 219/425] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:39.066234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a38a38-5c71-4ddd-8d3c-e127c65ff72e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.176563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:39.186151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 207/426] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:57:39.186609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a38a38-5c71-4ddd-8d3c-e127c65ff72e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:39.188372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deee2524-acca-458d-8585-e2195d1eea4c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.188950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deee2524-acca-458d-8585-e2195d1eea4c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.189446 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deee2524-acca-458d-8585-e2195d1eea4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.208526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-deee2524-acca-458d-8585-e2195d1eea4c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:39.210020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-deee2524-acca-458d-8585-e2195d1eea4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:39.225011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:39.226105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.226393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.226673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.228324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:39.230641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:39.236456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 220/427] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:39.237441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.283622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:39.297961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 208/428] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:39.298626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deee2524-acca-458d-8585-e2195d1eea4c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.363356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a24a459-3aae-44bc-bde3-525d0c8aba88 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:39.410245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:39.411111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.411396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.411670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] PATH_INFO: `/v3/projects/68b3ea181434496b91ea5053896483e8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.412135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deee2524-acca-458d-8585-e2195d1eea4c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:39.413014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] RBAC: Authorizing `identity:get_project(project_id=68b3ea181434496b91ea5053896483e8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:39.415691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:39.418363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 221/429] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/projects/68b3ea181434496b91ea5053896483e8 => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:39.419573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.452967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:39.463229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:39.464219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.464651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.465043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.466337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:39.468445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:39.472454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 209/430] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:39.473515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.543515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accbd742-dc9c-416a-a08c-9a301d602bc8 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:39.560620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:39.585174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:39.586266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.586562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.586845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.598553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:39.607602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:39.608522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:39.609331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 222/431] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:39.610067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.616152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3234c96e-e7a5-4c34-ad78-404706b98373 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:39.662723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:39.664596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.665021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.665688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.667399 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:39.669579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:39.672173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 210/432] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:39.673795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.712501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:39.734223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d83543-cdf0-413c-9f8e-5351745ae385 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:39.779829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:39.781382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.781858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.782537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.783789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=68b3ea181434496b91ea5053896483e8) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:39.784705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:39.786613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:39.790719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc13f98-d9f6-4de6-ac92-57ec89a8a06b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:39.795497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 223/433] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 611 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:39.801547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.837075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:39.859661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:39.860471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:39.860869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:39.861249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:39.862403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] RBAC: Adding query filter params (name=member) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:39.862862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:39.866317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:39.891780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 211/434] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:39.897070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:39.993074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3c4232-d80b-44c5-8936-1038d1153392 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.029510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b2ac993-c6ec-49f7-bbe3-388753f2508f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.030178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ac993-c6ec-49f7-bbe3-388753f2508f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.030645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ac993-c6ec-49f7-bbe3-388753f2508f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.030933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2ac993-c6ec-49f7-bbe3-388753f2508f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.031777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2ac993-c6ec-49f7-bbe3-388753f2508f None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=68b3ea181434496b91ea5053896483e8) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:40.032097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2ac993-c6ec-49f7-bbe3-388753f2508f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:40.033737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b2ac993-c6ec-49f7-bbe3-388753f2508f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:40.039857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 224/435] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=68b3ea181434496b91ea5053896483e8&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 611 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:40.041392 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e0fd8-0c7e-4ecb-916f-59defe0d359a None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.041604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e0fd8-0c7e-4ecb-916f-59defe0d359a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.041888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e0fd8-0c7e-4ecb-916f-59defe0d359a None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.043461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 225/436] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:40.044906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95c9a2a-1d39-474f-b00f-4121a401b0f2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.058086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d8eab3-22e5-41f0-9523-1bee6ec95ec2 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.058749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d8eab3-22e5-41f0-9523-1bee6ec95ec2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.059259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d8eab3-22e5-41f0-9523-1bee6ec95ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.083532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e2d8eab3-22e5-41f0-9523-1bee6ec95ec2 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:40.085223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e2d8eab3-22e5-41f0-9523-1bee6ec95ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:40.093432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.094163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.094445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.094718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.103721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.170153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.177259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 226/437] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:40.188048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:40.189576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:40.202309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d8eab3-22e5-41f0-9523-1bee6ec95ec2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:40.320092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 212/438] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:57:40.326909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:40.354815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d8eab3-22e5-41f0-9523-1bee6ec95ec2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.395544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.396913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.396913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.396913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.411904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:40.422175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:40.423055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:40.423835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 227/439] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:40.429351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:40.443680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.447364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.460295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac9db7-921c-4a9a-b2d0-7caeb64f6913 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.505317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.506379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.506769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.507149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.508371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:40.511024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:40.513670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 213/440] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:40.519735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:40.561255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c3302-2d2b-4ef3-8e3d-bd8bd70384d5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.601644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c4915e-ca0b-419a-8e09-6a9c6c5ad1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.602251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c4915e-ca0b-419a-8e09-6a9c6c5ad1db None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.603025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c4915e-ca0b-419a-8e09-6a9c6c5ad1db None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.603312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c4915e-ca0b-419a-8e09-6a9c6c5ad1db None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.604077 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c4915e-ca0b-419a-8e09-6a9c6c5ad1db None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:40.606376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c4915e-ca0b-419a-8e09-6a9c6c5ad1db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:40.606376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c4915e-ca0b-419a-8e09-6a9c6c5ad1db None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:40.615845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 228/441] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:40.617405 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa68fff0-6c9c-4845-9c39-825b64ab0f63 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.617765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa68fff0-6c9c-4845-9c39-825b64ab0f63 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.618054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa68fff0-6c9c-4845-9c39-825b64ab0f63 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.619433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 229/442] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:40.622586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa68fff0-6c9c-4845-9c39-825b64ab0f63 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:40.642791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10729b2f-aa4d-4d70-980f-96258da86878 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.675835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d09518-0c1e-4950-b193-e6dfcc1401d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.676712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d09518-0c1e-4950-b193-e6dfcc1401d0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.677000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d09518-0c1e-4950-b193-e6dfcc1401d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.677277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d09518-0c1e-4950-b193-e6dfcc1401d0 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.678456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d09518-0c1e-4950-b193-e6dfcc1401d0 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:40.679897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d09518-0c1e-4950-b193-e6dfcc1401d0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:40.681547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 214/443] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:40.682667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335429a-357b-4895-8493-31b33bc9444e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.682952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335429a-357b-4895-8493-31b33bc9444e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.683760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c335429a-357b-4895-8493-31b33bc9444e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.697289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c335429a-357b-4895-8493-31b33bc9444e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:40.701360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c335429a-357b-4895-8493-31b33bc9444e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:40.718775 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.730633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa68fff0-6c9c-4845-9c39-825b64ab0f63 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.765202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.765839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.766121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.766663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.769982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 215/444] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:40.771455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c335429a-357b-4895-8493-31b33bc9444e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:40.835083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:40.836579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:40.872817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c335429a-357b-4895-8493-31b33bc9444e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:40.903449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.907734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:40.908378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:40.908688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:40.908960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:40.909796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=eefd36f11cba4b2eac18767b1b872d97, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:40.910105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:40.911534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:40.917276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 216/445] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=eefd36f11cba4b2eac18767b1b872d97 => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:40.918071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:40.938604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:40.953325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 230/446] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:57:40.959938 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.030044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658a76bd-39ef-4a18-a52e-b202fb359fa5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.068695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.069504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.069783 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.070063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.082671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:41.084972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4d4f5d-fd32-441e-8c30-6df5a172ef2a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:41.090421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:41.091323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:41.093225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 217/447] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:41.097678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.125585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.126423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.126705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.126983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] PATH_INFO: `/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.128179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] RBAC: Authorizing `identity:get_project(project_id=bbbd945e8b7c45298ee54ed9a2ac0fd6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:41.129705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:41.131515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 231/448] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:40 2022] GET /identity/v3/projects/bbbd945e8b7c45298ee54ed9a2ac0fd6 => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:41.137719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.184782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:41.190283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:41.215934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486a629-b161-4c42-8c35-b4743504e637 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.258648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.259318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.259602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.259876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.260740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:41.261046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:41.262500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:41.263406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7857123c-61cb-4d1d-ba92-89f359dfe148 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.272418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 218/449] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:41.278421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.300821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.301678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.301965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.302247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] PATH_INFO: `/v3/groups/e0b51cfe57224b8eb97896476baa3bc4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.303422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] RBAC: Authorizing `identity:get_group(group_id=e0b51cfe57224b8eb97896476baa3bc4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:41.304934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:41.306565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 232/450] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:41 2022] GET /identity/v3/groups/e0b51cfe57224b8eb97896476baa3bc4 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:41.312201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.384891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008bef09-34c8-4d7d-bbae-fe9922b923ac None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.408556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c146f36-6712-49dd-83d7-c4de542c8fb3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.415543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:41.417549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.418236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.418515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.418786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.428940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:41.442924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.443491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.443779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.444057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.445074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] RBAC: Adding query filter params (group.id=e0b51cfe57224b8eb97896476baa3bc4, role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:41.445389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:41.446843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:41.453094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 233/451] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:41 2022] GET /identity/v3/role_assignments?group.id=e0b51cfe57224b8eb97896476baa3bc4&scope.project.id=bbbd945e8b7c45298ee54ed9a2ac0fd6&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 611 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:41.490058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:41.491530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:41.594141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 219/452] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:57:41.599761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.703938 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c579b923-2991-4bf7-bc15-e4cefc45bb95 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.737301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.738168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.738458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.738737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] PATH_INFO: `/v3/projects/ba1e700aae1b46039e859b8f609bb931` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.739926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] RBAC: Authorizing `identity:get_project(project_id=ba1e700aae1b46039e859b8f609bb931)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:41.741409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:41.743109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 234/453] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:57:41 2022] GET /identity/v3/projects/ba1e700aae1b46039e859b8f609bb931 => generated 344 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:41.748845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.794052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:41.850904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5eb6bf-e5d0-4872-8ecf-87d38c75fb77 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:41.883267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6e72d-e6fd-4f1f-815c-a492c1ec794c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:41.884117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6e72d-e6fd-4f1f-815c-a492c1ec794c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:41.884421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6e72d-e6fd-4f1f-815c-a492c1ec794c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:41.884706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6e72d-e6fd-4f1f-815c-a492c1ec794c None admin] PATH_INFO: `/v3/groups/772e5a0905bf4f68ba80c5614360ee22` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:41.885857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6e72d-e6fd-4f1f-815c-a492c1ec794c None admin] RBAC: Authorizing `identity:get_group(group_id=772e5a0905bf4f68ba80c5614360ee22)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:41.887855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6e72d-e6fd-4f1f-815c-a492c1ec794c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:41.889498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 220/454] 10.222.0.20 () {56 vars in 1180 bytes} [Tue Jul 5 09:57:41 2022] GET /identity/v3/groups/772e5a0905bf4f68ba80c5614360ee22 => generated 229 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:41.895534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:41.935980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:41.992323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb5cbfd-d3d1-4452-a1ff-794ac78879ed None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:42.024879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affb2a2e-e37b-4e04-ac7d-92fbe395c5ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:42.025521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affb2a2e-e37b-4e04-ac7d-92fbe395c5ff None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:42.025803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affb2a2e-e37b-4e04-ac7d-92fbe395c5ff None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:42.026078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affb2a2e-e37b-4e04-ac7d-92fbe395c5ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:42.026905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affb2a2e-e37b-4e04-ac7d-92fbe395c5ff None admin] RBAC: Adding query filter params (group.id=772e5a0905bf4f68ba80c5614360ee22, role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=ba1e700aae1b46039e859b8f609bb931) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:42.027700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affb2a2e-e37b-4e04-ac7d-92fbe395c5ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:42.029146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affb2a2e-e37b-4e04-ac7d-92fbe395c5ff None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:42.034690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 235/455] 10.222.0.20 () {56 vars in 1376 bytes} [Tue Jul 5 09:57:41 2022] GET /identity/v3/role_assignments?group.id=772e5a0905bf4f68ba80c5614360ee22&scope.project.id=ba1e700aae1b46039e859b8f609bb931&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 611 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:44.356244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f2ee8-564f-4d9e-92fd-a33dc18e5de3 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:44.356818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f2ee8-564f-4d9e-92fd-a33dc18e5de3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:44.356818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316f2ee8-564f-4d9e-92fd-a33dc18e5de3 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:44.358482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 221/456] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:44.363813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc8d812-8a61-4073-9b40-4ee11f7dc5aa None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:44.364654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc8d812-8a61-4073-9b40-4ee11f7dc5aa None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:44.365130 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc8d812-8a61-4073-9b40-4ee11f7dc5aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:44.389372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-edc8d812-8a61-4073-9b40-4ee11f7dc5aa None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:44.390924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-edc8d812-8a61-4073-9b40-4ee11f7dc5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:44.481900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 236/457] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:44.521281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f2ee8-564f-4d9e-92fd-a33dc18e5de3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:44.654106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316f2ee8-564f-4d9e-92fd-a33dc18e5de3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:44.664224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:44.700352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aa8a6e-3293-48ed-a80d-5c87127e17c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:44.701180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aa8a6e-3293-48ed-a80d-5c87127e17c9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:44.701466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aa8a6e-3293-48ed-a80d-5c87127e17c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:44.701742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aa8a6e-3293-48ed-a80d-5c87127e17c9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:44.712331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12aa8a6e-3293-48ed-a80d-5c87127e17c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:44.713801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12aa8a6e-3293-48ed-a80d-5c87127e17c9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-12aa8a6e-3293-48ed-a80d-5c87127e17c9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:44.742391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:44.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:44.744165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:44.744165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:44.744165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:44.744165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:44.744165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 222/458] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:44.753526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc8d812-8a61-4073-9b40-4ee11f7dc5aa None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:44.810599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e1215a-eae2-4810-ae82-560373669783 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:44.810947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e1215a-eae2-4810-ae82-560373669783 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:44.811162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e1215a-eae2-4810-ae82-560373669783 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:44.812810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 223/459] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:57:44.817105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:44.843266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84223b0-82c7-49ff-8343-1454d2b38f1a None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:44.843580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84223b0-82c7-49ff-8343-1454d2b38f1a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:44.843829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84223b0-82c7-49ff-8343-1454d2b38f1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:44.873478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f84223b0-82c7-49ff-8343-1454d2b38f1a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:44.874699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f84223b0-82c7-49ff-8343-1454d2b38f1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:44.904703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc8d812-8a61-4073-9b40-4ee11f7dc5aa None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:44.945969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:44.947011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:44.947709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:44.949497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:44.950651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:44.951903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:44.953731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:44.978877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 237/460] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:44.984410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.034707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 224/461] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:45.041989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55189fa-eb50-4f04-b0fd-b650bc749462 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.042273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55189fa-eb50-4f04-b0fd-b650bc749462 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.042573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55189fa-eb50-4f04-b0fd-b650bc749462 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.044159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 225/462] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:57:45.048601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2e06f-eba6-437e-a5a8-3e79393f84a0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.048884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2e06f-eba6-437e-a5a8-3e79393f84a0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.049163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd2e06f-eba6-437e-a5a8-3e79393f84a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.071880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9cd2e06f-eba6-437e-a5a8-3e79393f84a0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:45.074500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9cd2e06f-eba6-437e-a5a8-3e79393f84a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:45.117180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:45.126537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b43b27-a9b0-431e-8ce1-052b7c872a7a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.151538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 226/463] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:45.153158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd2e06f-eba6-437e-a5a8-3e79393f84a0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.163106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334c5dd9-08b5-412a-84b1-7f05b6203430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.163715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c5dd9-08b5-412a-84b1-7f05b6203430 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.164002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c5dd9-08b5-412a-84b1-7f05b6203430 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.164282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334c5dd9-08b5-412a-84b1-7f05b6203430 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.165008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334c5dd9-08b5-412a-84b1-7f05b6203430 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.166477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334c5dd9-08b5-412a-84b1-7f05b6203430 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:45.183621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:45.192532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 238/464] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:44 2022] POST /identity/v3/users => generated 246 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:57:45.193932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dfe28d-b1fe-44c1-b550-35100fae0c17 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.194226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dfe28d-b1fe-44c1-b550-35100fae0c17 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.194505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dfe28d-b1fe-44c1-b550-35100fae0c17 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.197041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 239/465] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:45.197041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dfe28d-b1fe-44c1-b550-35100fae0c17 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.282115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd2e06f-eba6-437e-a5a8-3e79393f84a0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.318681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dfe28d-b1fe-44c1-b550-35100fae0c17 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.321447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee7c59d-8a0a-4438-bc2d-91f076851eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.322584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee7c59d-8a0a-4438-bc2d-91f076851eda None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.323081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee7c59d-8a0a-4438-bc2d-91f076851eda None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.323580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee7c59d-8a0a-4438-bc2d-91f076851eda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.337299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee7c59d-8a0a-4438-bc2d-91f076851eda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.339432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee7c59d-8a0a-4438-bc2d-91f076851eda None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:45.358018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.358831 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.359119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.359400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-6ee7c59d-8a0a-4438-bc2d-91f076851eda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:45.361079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:45.362221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:45.363141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:45.363141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:45.363141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:45.363141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:45.370090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 227/466] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:45.370898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.372229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a8e2c-372d-4bb6-bbdf-2b2bc707b77d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.372688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a8e2c-372d-4bb6-bbdf-2b2bc707b77d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.373106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a8e2c-372d-4bb6-bbdf-2b2bc707b77d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.373215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:45.383235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:45.390804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b92a8e2c-372d-4bb6-bbdf-2b2bc707b77d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:45.392347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b92a8e2c-372d-4bb6-bbdf-2b2bc707b77d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:45.394871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:45.395729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:45.397076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:45.397076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:45.397076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:45.397076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:45.397076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 240/467] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:45.402906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.473423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 228/468] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:45.474312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a8e2c-372d-4bb6-bbdf-2b2bc707b77d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.477124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:45.528954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab8b93c-620f-4871-ac48-59d91b67bf2a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.578872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.580144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.581050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.581840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.583030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:45.583922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.585679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:45.589134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a8e2c-372d-4bb6-bbdf-2b2bc707b77d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.602646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 241/469] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:45.604100 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.631220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.631962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.632247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.632581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.633348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:45.633656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.635047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:45.655355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 229/470] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:45.656041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.684099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:45.732237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbba546f-9940-4c4f-b051-46bdc6042624 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.733955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:45.769227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.769901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.770189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.770468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.782364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.783837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:45.784672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac60d1f1-6dc3-4be3-8023-56762bc1dd1b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:45.802086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:45.802975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:45.804260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:45.804260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:45.804260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:45.804260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:45.804260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 242/471] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:45.804260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.828601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.829286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.829570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.829849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.830723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.832180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:45.857568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 230/472] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:45 2022] POST /identity/v3/users => generated 248 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:45.858809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:45.947998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788a6b4a-a94b-47d5-98e5-e728c701897c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.962136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:45.983472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae4d178-18bd-4a17-8c87-010bda25d85c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:45.989502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380f4d1d-5edc-4a9b-8354-b60fd3a302d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:45.990425 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380f4d1d-5edc-4a9b-8354-b60fd3a302d8 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:45.990853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380f4d1d-5edc-4a9b-8354-b60fd3a302d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:45.991357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380f4d1d-5edc-4a9b-8354-b60fd3a302d8 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:45.992296 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380f4d1d-5edc-4a9b-8354-b60fd3a302d8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:45.994053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380f4d1d-5edc-4a9b-8354-b60fd3a302d8 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:46.008790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:46.018931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 243/473] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:45 2022] POST /identity/v3/users => generated 248 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:46.020662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c781d0-e080-4277-8385-f38808277c2b None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:46.021229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c781d0-e080-4277-8385-f38808277c2b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:46.021690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c781d0-e080-4277-8385-f38808277c2b None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:46.023592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 244/474] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:46.030406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:46.030406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:46.030406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:46.030406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:46.030825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:46.030825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:46.030825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:46.032815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dbf0f7-5354-4679-b861-3d1d6d79dc2a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:46.033334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dbf0f7-5354-4679-b861-3d1d6d79dc2a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:46.033818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dbf0f7-5354-4679-b861-3d1d6d79dc2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:46.050550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 231/475] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:46.051899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:46.058953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b9dbf0f7-5354-4679-b861-3d1d6d79dc2a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:46.060654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b9dbf0f7-5354-4679-b861-3d1d6d79dc2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:46.143920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 245/476] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:46.174870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82a2cd9-8b5f-4fdd-8cef-76c0e7dfc47b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:46.179290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dbf0f7-5354-4679-b861-3d1d6d79dc2a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:46.221908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:46.222939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:46.223362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:46.223799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:46.224872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:46.227445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:46.256072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 232/477] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:46 2022] POST /identity/v3/users => generated 249 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:46.306542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:46.313071 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dbf0f7-5354-4679-b861-3d1d6d79dc2a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:46.352875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:46.353879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:46.354562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:46.356004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:46.356818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:46.358469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:46.367294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.sql.core [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 05 09:57:46.367294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 05 09:57:46.367294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [parameters: {'id': '1b79393e2bbd4e11966eb50006223227', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 05 09:57:46.367294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80269) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result = self._query(query) Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application conn.query(q) Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result.read() Jul 05 09:57:46.385919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return ref.to_dict() Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 05 09:57:46.387364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application next(self.gen) Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application yield resource Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application next(self.gen) Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.session.rollback() Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.force_reraise() Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise self.value Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application session.commit() Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 05 09:57:46.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._prepare_impl() Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.session.flush() Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._flush(objects) Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application compat.raise_( Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application flush_context.execute() Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rec.execute(self) Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 05 09:57:46.389729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 05 09:57:46.390625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result = self._query(query) Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application conn.query(q) Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result.read() Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 05 09:57:46.391421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [parameters: {'id': '1b79393e2bbd4e11966eb50006223227', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:46.393766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 05 09:57:46.395661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:46.401849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 246/478] 10.222.0.20 () {60 vars in 1131 bytes} [Tue Jul 5 09:57:46 2022] POST /identity/v3/roles => generated 144 bytes in 209 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:46.402530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:46.472560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:46.545696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4374aa56-bd8f-4132-ac53-7e3207564d44 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:46.585231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182aa800-f333-42ff-bbed-0ee1e1bf56fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:46.586086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182aa800-f333-42ff-bbed-0ee1e1bf56fb None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:46.586378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182aa800-f333-42ff-bbed-0ee1e1bf56fb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:46.586659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182aa800-f333-42ff-bbed-0ee1e1bf56fb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:46.602018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182aa800-f333-42ff-bbed-0ee1e1bf56fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-182aa800-f333-42ff-bbed-0ee1e1bf56fb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:46.610849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:46.611711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:46.612812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 233/479] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:57:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:46.617653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:46.728315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb33d65-1af1-4a74-b2e2-f704224c6faa None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:46.769742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1235bd95-0293-4af8-bed7-fa094de44153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:46.770419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1235bd95-0293-4af8-bed7-fa094de44153 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:46.770707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1235bd95-0293-4af8-bed7-fa094de44153 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:46.770986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1235bd95-0293-4af8-bed7-fa094de44153 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:46.771745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1235bd95-0293-4af8-bed7-fa094de44153 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:46.772056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1235bd95-0293-4af8-bed7-fa094de44153 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:46.773498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1235bd95-0293-4af8-bed7-fa094de44153 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:46.782945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 247/480] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:47.567000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aee47a-1423-457d-9be3-82abda2ef189 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:47.567539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aee47a-1423-457d-9be3-82abda2ef189 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:47.567758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aee47a-1423-457d-9be3-82abda2ef189 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:47.569442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 234/481] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:47.574760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78184c0-92d0-4734-8a75-feaecb0db618 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:47.575151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78184c0-92d0-4734-8a75-feaecb0db618 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:47.575311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78184c0-92d0-4734-8a75-feaecb0db618 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:47.596446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c78184c0-92d0-4734-8a75-feaecb0db618 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:47.597650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c78184c0-92d0-4734-8a75-feaecb0db618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:47.625167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:47.686929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 248/482] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:47.727060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aee47a-1423-457d-9be3-82abda2ef189 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:47.842840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:47.857985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aee47a-1423-457d-9be3-82abda2ef189 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:47.899856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:47.900686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:47.900971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:47.901247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:47.917388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:47.926186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:47.927054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:47.927847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 235/483] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:47.936424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78184c0-92d0-4734-8a75-feaecb0db618 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:48.100533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78184c0-92d0-4734-8a75-feaecb0db618 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:48.145131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:48.145809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.146095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.146377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.147136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:48.147444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:48.148873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:48.157781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 249/484] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:48.163332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:48.307366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac2aa9c-449a-49e7-87bb-46c7ec7d1d6c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:48.358997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480d1354-4412-477c-b972-07d1a4ddb8c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:48.360117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480d1354-4412-477c-b972-07d1a4ddb8c7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.360607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480d1354-4412-477c-b972-07d1a4ddb8c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.361064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480d1354-4412-477c-b972-07d1a4ddb8c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.376664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480d1354-4412-477c-b972-07d1a4ddb8c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:48.378400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480d1354-4412-477c-b972-07d1a4ddb8c7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-480d1354-4412-477c-b972-07d1a4ddb8c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:48.401063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:48.401931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:48.403320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:48.403320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:48.403320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:48.403320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:48.413522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:48.416677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 236/485] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:48.436455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:48.563783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2b5cb6-9711-462b-93f8-8a2af6733ac8 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:48.591434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e499007-62fc-4076-beaa-f5ba6cb69671 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.592356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e499007-62fc-4076-beaa-f5ba6cb69671 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.592897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e499007-62fc-4076-beaa-f5ba6cb69671 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.594725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 237/486] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:57:48.600796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdc2644-7129-487d-b7f3-a0a0e6e78064 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.601652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdc2644-7129-487d-b7f3-a0a0e6e78064 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.602139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdc2644-7129-487d-b7f3-a0a0e6e78064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.621344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:48.625027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8cdc2644-7129-487d-b7f3-a0a0e6e78064 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:48.625981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:48.626663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.626954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.627229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.627764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8cdc2644-7129-487d-b7f3-a0a0e6e78064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:48.627951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:48.628263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:48.629666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:48.653598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 250/487] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:48.654914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:48.718349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 238/488] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:48.724413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491a0f9-7158-4f38-ab49-8350434d019b None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.724413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491a0f9-7158-4f38-ab49-8350434d019b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.724413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f491a0f9-7158-4f38-ab49-8350434d019b None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.724413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 239/489] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:48 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:48.732806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a029f4-bf43-40bd-abec-95d6355feb25 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.733025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a029f4-bf43-40bd-abec-95d6355feb25 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.733242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a029f4-bf43-40bd-abec-95d6355feb25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.755635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-52a029f4-bf43-40bd-abec-95d6355feb25 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:48.757929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-52a029f4-bf43-40bd-abec-95d6355feb25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:48.783645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:48.810731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313cd27f-66fa-49d1-b91b-8ca59a6925d3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:48.832472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 240/490] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:48.834386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a029f4-bf43-40bd-abec-95d6355feb25 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:48.865852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:48.867125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:48.867662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:48.868139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] PATH_INFO: `/v3/users/nova` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:48.876364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:48.878151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:48.894977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:48.895943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:57:48.896892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:48.899238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 251/491] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:57:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:48.904422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:48.923006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:48.969940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a029f4-bf43-40bd-abec-95d6355feb25 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.008595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.009906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.010368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.010816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.016617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7514ac84-5351-43ab-a84e-3f24f4c6993c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.023680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:49.033177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:49.034065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:49.034861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:49.042267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 241/492] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:49.043380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.052484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.053290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.053578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.053857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.056710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.065793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:49.074605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:49.075474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:49.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:49.080489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 252/493] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:49.081428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.160495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a384b-e085-47e4-9119-4088a0efad7c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.186711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.191091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b295c74b-cf2d-4401-9636-ea355bc4fb6b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.195562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.196240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.196570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.196878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.197696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:49.198006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:49.200017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:49.226570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd229279-7665-499b-a06d-87d972f871f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.227238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd229279-7665-499b-a06d-87d972f871f3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.227525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd229279-7665-499b-a06d-87d972f871f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.227803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd229279-7665-499b-a06d-87d972f871f3 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.228581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd229279-7665-499b-a06d-87d972f871f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:49.228891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd229279-7665-499b-a06d-87d972f871f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:49.230301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd229279-7665-499b-a06d-87d972f871f3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:49.236865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 242/494] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:49.237705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.243895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 253/495] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:49.245474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f87c6-6435-4a9a-8ec7-9cb693488a64 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.245977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f87c6-6435-4a9a-8ec7-9cb693488a64 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.246413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00f87c6-6435-4a9a-8ec7-9cb693488a64 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.247942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 254/496] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:49.248838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f87c6-6435-4a9a-8ec7-9cb693488a64 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.316029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.345868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e61afe-655c-4db1-9aab-ad3038872f52 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.361923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00f87c6-6435-4a9a-8ec7-9cb693488a64 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.380381 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c60fe5-b26c-43ae-af8f-9fabd3bdcb59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.381086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c60fe5-b26c-43ae-af8f-9fabd3bdcb59 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.381371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c60fe5-b26c-43ae-af8f-9fabd3bdcb59 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.381650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c60fe5-b26c-43ae-af8f-9fabd3bdcb59 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.382404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c60fe5-b26c-43ae-af8f-9fabd3bdcb59 None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:49.382712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c60fe5-b26c-43ae-af8f-9fabd3bdcb59 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:49.384112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c60fe5-b26c-43ae-af8f-9fabd3bdcb59 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:49.392254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 243/497] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:49.393455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738778fd-39c5-40be-b759-3a75af473928 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.393744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738778fd-39c5-40be-b759-3a75af473928 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.394023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738778fd-39c5-40be-b759-3a75af473928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.399864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5685c8ab-8be2-4aa3-b63a-69129ece6cf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.400548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685c8ab-8be2-4aa3-b63a-69129ece6cf8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.400837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685c8ab-8be2-4aa3-b63a-69129ece6cf8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.401117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685c8ab-8be2-4aa3-b63a-69129ece6cf8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.411232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-738778fd-39c5-40be-b759-3a75af473928 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:49.412382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-738778fd-39c5-40be-b759-3a75af473928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:49.453894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.468016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5685c8ab-8be2-4aa3-b63a-69129ece6cf8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:49.469528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5685c8ab-8be2-4aa3-b63a-69129ece6cf8 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:49.478881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 244/498] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:49.479680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738778fd-39c5-40be-b759-3a75af473928 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.529264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.570982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 255/499] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:49.572348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957aa751-abdc-4932-ab02-4ffd3c8f8fbc None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.572684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957aa751-abdc-4932-ab02-4ffd3c8f8fbc None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.572966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957aa751-abdc-4932-ab02-4ffd3c8f8fbc None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.574278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 256/500] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:49.574976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957aa751-abdc-4932-ab02-4ffd3c8f8fbc None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.584066 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738778fd-39c5-40be-b759-3a75af473928 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.618911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.619691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.619969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.620242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.630897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:49.632352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:49.650210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:49.651070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:49.651912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:49.651912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:49.651912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:49.651912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:49.651912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 245/501] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:49.652179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.675949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957aa751-abdc-4932-ab02-4ffd3c8f8fbc None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.682576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.711563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.712344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.712654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.712929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.722603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:49.724056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:49.741550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:49.742380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:49.743215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:49.743215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:49.743215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:49.743215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:49.743215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 257/502] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:49.743470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.757022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93eefe34-214a-4ba5-93ef-d688a44d87c8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.759138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.789666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731d04b2-3ed0-4a50-af74-86142fc1c32c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.790452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d04b2-3ed0-4a50-af74-86142fc1c32c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.790739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d04b2-3ed0-4a50-af74-86142fc1c32c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.791016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731d04b2-3ed0-4a50-af74-86142fc1c32c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.801695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731d04b2-3ed0-4a50-af74-86142fc1c32c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-731d04b2-3ed0-4a50-af74-86142fc1c32c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:49.809967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:49.810884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:49.811682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 246/503] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:49.812524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279d06d-643e-419c-9ca1-931d96ac3656 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.812703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279d06d-643e-419c-9ca1-931d96ac3656 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.812983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279d06d-643e-419c-9ca1-931d96ac3656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.829426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4279d06d-643e-419c-9ca1-931d96ac3656 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:49.830625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4279d06d-643e-419c-9ca1-931d96ac3656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:49.846306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e411c2b-7941-4c18-a370-bac2ed9467ba None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:49.879534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:49.880423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:49.880712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:49.880993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:49.891417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:49.893168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:49.896278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 247/504] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:49.897131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4279d06d-643e-419c-9ca1-931d96ac3656 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:49.910355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:49.911197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:49.912041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:49.912041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:49.912041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:49.912041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:49.912041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 258/505] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:49.912307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:49.929069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:49.975116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.010715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4279d06d-643e-419c-9ca1-931d96ac3656 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.015306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ac43e5-aad2-402f-95bb-7bef019bc58e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.044683 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.045345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.045705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.045904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.046628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.046935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.048327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.049636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.050286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.050571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.050846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.051557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.051860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.053244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.060011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 248/506] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:50.060827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.070237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 259/507] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:50.070933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.137509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.159457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c66a3d7-fb2a-4301-9371-0f01ade8af67 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.169744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7dd7b9-97f0-4709-8ba3-39da65035ac9 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.192406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.192749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.193014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.193652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.193969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.194733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.195081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.196488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.204334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 249/508] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:50.204911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.205198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.205640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.205931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.206211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.206937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.207249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.208740 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.225050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 260/509] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:50.225050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.305000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02345ef3-ef18-4fca-9670-d7b85f43051e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.321313 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839ba8a-d7c4-4ef6-9d7a-7bbef8e23d90 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.338253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.339034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.339315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.340125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.348512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.350865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.352656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.354785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.356153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.356468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.356748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.364046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.365496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:50.370045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:50.371004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:50.372039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:50.372039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:50.372039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:50.372039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:50.372039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 250/510] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:50.372039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:50.381137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:50.382199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 05 09:57:50.383262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:50.384529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 261/511] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/users/glance => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:50.385436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.428104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.470480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cbbe46-9631-4a1c-976f-9f1a9a10df4e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.483163 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff16f782-c98a-46d5-88e0-b4a20b2860a5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.503035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.503824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.504106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.504528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.512506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.513978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.516249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.517307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.517587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.517860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.528653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.530098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:50.530207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:50.531217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 251/512] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/users/cinder => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:50.532300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:50.550689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:50.551679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:50.552721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:50.552721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:50.552721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:50.552721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:50.552721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 262/513] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:50.552721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.587247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.633601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c59f095-fd55-45ce-92cf-f4cbd0afc4ea None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.648873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.656572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b294c26d-2480-41c4-a245-7d84a0df50be None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.667426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.668796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.669081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.669354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.675854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.677338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.689426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.690073 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.690357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.690663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.691377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.691760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:50.692315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:50.693296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:50.694557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:50.695090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.695170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 252/514] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:50.695170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.709843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 263/515] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:50.710644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.798296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d7fef2-c872-4d4f-8bf0-86d5e2f5515d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.806853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdb81da-a523-4429-b3fb-03b48db45649 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.811171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.832497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.833163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.833444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.833725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.834553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.834861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.836295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.840838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.841499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.841784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.842064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.842895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.843208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.844660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:50.853505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 253/516] 10.222.0.20 () {56 vars in 1137 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:50.854129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.883639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 264/517] 10.222.0.20 () {56 vars in 1137 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:50.884513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:50.899668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:50.955668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90e7d44-05d0-49e2-aced-7499c29736e4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.986265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba317545-7619-42a1-99d7-71b8523a1323 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:50.989416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:50.990078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:50.990361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:50.990636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:50.991355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:50.991656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:50.993071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.004540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 254/518] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:51.005305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.023111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.023765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.024049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.024323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.025131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:51.025440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.026853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.034556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.044910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 265/519] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:51.045716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.106939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41325090-68ff-452c-b27c-27d9a8097e27 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.114269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.140794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.141461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.141994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.142278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.142972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.144416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.145039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadb4f09-4db1-4ac3-8bca-e2146f754d8a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.163035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 255/520] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:57:51 2022] POST /identity/v3/users => generated 247 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:57:51.163827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.180481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.181391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.181729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.182012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.265360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.267124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.276109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.295615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224f74e4-b972-4b2b-a7be-05083c112096 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.337318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.337526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.337677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.337955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.348791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.388594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 266/521] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:51.391623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.418943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.420891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.492807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc7fd7c-9a59-4d35-90b8-596e00c428c7 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.513005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.528925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.529804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.530000 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.530281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.537103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 256/522] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:51.537103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.543549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.545057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:51.560592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:51.561671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 267/523] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/users/neutron => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:51.562564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.594507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.647247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c22fe0-d453-40f8-a48b-481668ff13b5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.674611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f51f2ca-8890-4bfb-a5e2-85fbcd5a43eb None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.684491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.685450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.685954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.686421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.687426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=f99a9fe4b40646d389c3ebf47358956c) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:51.687920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.689567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.695867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 257/524] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/role_assignments?user.id=f99a9fe4b40646d389c3ebf47358956c&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:51.696879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.724142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.725063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.725413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.725750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.738592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.740241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:51.760174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:51.761160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:51.762653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:51.762653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:51.762653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:51.762653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:51.762653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 268/525] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:51.762924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.768901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.832039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3980ad4-b8eb-4480-a185-103c46fb0218 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.873279 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.874088 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.874370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.874646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.887839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:51.889633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.890333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.912617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce18bfc-0e15-4536-9cca-03ba01e5a76c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:51.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:51.922571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:51.923406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:51.923406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:51.923406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:51.923406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:51.925794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 258/526] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:51.926714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:51.960096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:51.961222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:51.961779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:51.962066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:51.963142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:51.963780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:51.965762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:51.992498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 269/527] 10.222.0.20 () {56 vars in 1139 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:51.993520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.061550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe6d4b4-8255-47f2-a5ea-8f6000375780 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.063768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.106955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.107628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.107914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.108434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.109175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:52.109535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.111183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.120413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5212e916-f074-4eaf-8ac2-a14f8c017c17 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.124467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 259/528] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:52.125285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.158024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.159053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.159563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.160087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.161219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:52.161725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.163478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.176795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 270/529] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:52.178084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.181217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.251274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0645782-f6b7-4eb0-8ff1-7b126dd576c7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.293557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6a761-2b49-4eea-a868-953de5771207 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.293744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.294027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.294302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.327873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.336160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.337279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.337813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.338308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.346541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.348566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:52.364836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:52.365709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:52.366540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:52.367402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.368635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.369481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 271/530] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:52.371255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.438281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.476474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 260/531] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:57:52.477339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.482336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2d3791-e07f-4a46-94d9-3825ed9b4887 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.520432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.521671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.522159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.522681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.530163 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.531859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.541313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:52.549436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:52.550276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:52.551099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:52.553666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 272/532] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:52.554711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.582222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8347b70-c206-42f2-bd42-4f6941ef6a5d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.618163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.618654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.618943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.619223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.620000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:52.620319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.622393 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.640854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 261/533] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:52.641726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.664889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd48e12-1619-4258-a2f4-b45d96d4d86c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.689678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.704669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.705854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.706337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.706859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.719363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.721172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:52.740325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:52.741640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:52.742487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:52.742487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:52.742487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:52.742487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:52.744680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 273/534] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:52.745844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.750385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c0beb-b240-410c-af9c-0c0d950ff03a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.774906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.787643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.788352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.788665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.788989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.789798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:52.790109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.791546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.804047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 262/535] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:52.804906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.870182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273f0f49-8c2d-4fed-b5f3-a7c0382be2b2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.909153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e6a1f8-98fc-4418-ba68-d88e0a20288f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:52.909615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.910280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.910675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.910963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.911907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=90d9b9421d454465b8f99137cd182abf) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:52.912229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.913716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:52.920958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 274/536] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/role_assignments?user.id=90d9b9421d454465b8f99137cd182abf&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:52.922004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:52.974357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:52.978698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:52.979383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:52.979684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:52.979947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:52.981474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:52.981590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:52.982929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:53.008884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 263/537] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:53.014106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:53.049026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903bf8c6-b9ec-4e39-be46-1b11c548aeaf None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:53.091175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:53.092793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.093578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.094732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.095948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=1fc5b3da67de48cb83848168b16dd153) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:53.097593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:53.099494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:53.120207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 275/538] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:52 2022] GET /identity/v3/role_assignments?user.id=1fc5b3da67de48cb83848168b16dd153&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:53.129722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:53.214816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb12e08-e033-4c2a-87f3-45d9421eb6a8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:53.268050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c1cc79-c0fb-40de-95bb-b5619f5b054d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:53.269123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c1cc79-c0fb-40de-95bb-b5619f5b054d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.269553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c1cc79-c0fb-40de-95bb-b5619f5b054d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.269997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c1cc79-c0fb-40de-95bb-b5619f5b054d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.283885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c1cc79-c0fb-40de-95bb-b5619f5b054d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:53.285618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c1cc79-c0fb-40de-95bb-b5619f5b054d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-94c1cc79-c0fb-40de-95bb-b5619f5b054d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:53.303282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:53.304364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:53.305247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:53.307733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 264/539] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:53.313548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:53.403961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17296cc-89dc-41a9-8421-374d33e5cc5a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.404752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17296cc-89dc-41a9-8421-374d33e5cc5a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.405555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17296cc-89dc-41a9-8421-374d33e5cc5a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.407722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 265/540] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:53 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:53.415313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418c5128-132d-492b-ad94-ee324c2cd62c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.415888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418c5128-132d-492b-ad94-ee324c2cd62c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.416377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418c5128-132d-492b-ad94-ee324c2cd62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.437691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-418c5128-132d-492b-ad94-ee324c2cd62c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:53.439408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-418c5128-132d-492b-ad94-ee324c2cd62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:53.476289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cb695a-6d3e-48a1-a685-2c24c33b7116 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:53.478117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:53.541905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:53.542509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.542803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.543081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.543876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:53.544185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:53.545671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:53.545883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 266/541] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:53.567934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 276/542] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:53.573349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418c5128-132d-492b-ad94-ee324c2cd62c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:53.573467 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:53.627146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:53.725945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff7b72-3076-404e-af5b-a3ccf8468dae None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:53.750945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418c5128-132d-492b-ad94-ee324c2cd62c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:53.790834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8404c5-7f23-4b15-8cbc-fd2b84d43763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:53.791642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8404c5-7f23-4b15-8cbc-fd2b84d43763 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.791978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8404c5-7f23-4b15-8cbc-fd2b84d43763 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.792203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8404c5-7f23-4b15-8cbc-fd2b84d43763 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.795074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:53.796973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.797265 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.797547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.798397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=36ce9bd8ce6448b8b09fe9eb7b8f7c54) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:53.798711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:53.800176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:53.814990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 267/543] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:53 2022] GET /identity/v3/role_assignments?user.id=36ce9bd8ce6448b8b09fe9eb7b8f7c54&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 259 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:53.815652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8404c5-7f23-4b15-8cbc-fd2b84d43763 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-2f8404c5-7f23-4b15-8cbc-fd2b84d43763 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:53.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:53.825463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:53.826248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 277/544] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:53.832558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:53.897770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad0c42e-d70c-465a-b9ba-6fd31ce5c5b0 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.898404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad0c42e-d70c-465a-b9ba-6fd31ce5c5b0 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.898883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad0c42e-d70c-465a-b9ba-6fd31ce5c5b0 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.901397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 278/545] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:53.908433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f653db8-5894-493d-992a-903ed7480008 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:53.909492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f653db8-5894-493d-992a-903ed7480008 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:53.910296 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f653db8-5894-493d-992a-903ed7480008 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:53.949650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-7f653db8-5894-493d-992a-903ed7480008 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:53.951044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7f653db8-5894-493d-992a-903ed7480008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:53.988787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5223427-bfdf-4fe0-9aa9-3f50f5cd5a1f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:53.989435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:53.992833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:54.031165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:54.031984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:54.032408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:54.032787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:54.033734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:54.034178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:54.035787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:54.045946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 268/546] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:54.051324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:54.091037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 279/547] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:54.125210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f653db8-5894-493d-992a-903ed7480008 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:54.180255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440f7b33-6fa4-4c66-86fc-674a133b56df None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:54.223325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:54.224475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:54.224925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:54.225344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:54.245497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:54.250400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:54.278463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:54.279523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:54.280620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:54.280620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:54.280620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:54.280620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:54.288483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 269/548] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:54.295780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:54.311831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:54.330459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f653db8-5894-493d-992a-903ed7480008 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:54.364992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:54.365790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:54.366063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:54.366350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:54.377486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:54.385164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:54.386352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:54.388763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:54.391730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 280/549] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:54.392932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:54.413813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d1155f-a0c7-4b4f-a302-49f3dfb99abd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:54.420786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:54.451883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:54.452960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:54.453441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:54.453893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:54.454852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:54.455335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:54.456986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:54.471418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 270/550] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:54.476971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:54.542927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ef8195-2db8-4b25-8f93-247ea35473ea None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:54.599300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:54.600270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:54.600908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:54.601833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:54.603018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:54.603507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:54.605958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:54.616867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:54.631691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 281/551] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:54.632627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:54.678423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0960c-414b-4ccb-89ec-0fe9566d7ca1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:54.736149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:54.736969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:54.737249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:54.737518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:54.747011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:54.748674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:54.764989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:54.766412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:54.768237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 271/552] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/users/swift => generated 83 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:54.777823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:54.784747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:54.808114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b29182-45b7-44c2-9c0c-fe0e5fd2c947 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:54.846419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:54.847633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:54.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:54.848699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:54.860836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:54.862682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:54.881649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:54.882528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:54.883419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:54.883419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:54.883419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:54.883419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:54.885992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 282/553] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:54.894375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.031552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c77433-fa4b-45d5-bb5e-5f1ea9f3505e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:55.077703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932af334-ed57-4e73-9f59-8d5cd9300631 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:55.087534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:55.105704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:55.109100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:55.109803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.110099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.110386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.111559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:55.111797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:55.113332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:55.113442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:55.114100 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.114383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.114658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.115384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:55.115689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:55.117084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:55.130230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 283/554] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:55.135538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.140771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 272/555] 10.222.0.20 () {56 vars in 1135 bytes} [Tue Jul 5 09:57:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:55.146692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.242124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47dd1dd-ad84-419e-835d-07cfe9c6138f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:55.277867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:55.278666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.278946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.279244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.287336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:55.288990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:55.305961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:55.307015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:55.308997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 284/556] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:57:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:57:55.313640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.357943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee388196-7bf1-4582-8f59-66e4c61e9390 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:55.408820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:55.438009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe586f9-c781-4803-976f-c01a70ac3443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:55.438944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe586f9-c781-4803-976f-c01a70ac3443 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.439381 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe586f9-c781-4803-976f-c01a70ac3443 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.439789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe586f9-c781-4803-976f-c01a70ac3443 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.502355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ceb328-afdf-44ac-a343-2f5fd71be28e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:55.507959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:55.523869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe586f9-c781-4803-976f-c01a70ac3443 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:55.525705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe586f9-c781-4803-976f-c01a70ac3443 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:55.546984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b2eb8b-e6e7-46f5-a131-aad57f81fa2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:55.547659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b2eb8b-e6e7-46f5-a131-aad57f81fa2b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.547948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b2eb8b-e6e7-46f5-a131-aad57f81fa2b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.548229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b2eb8b-e6e7-46f5-a131-aad57f81fa2b None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.549131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b2eb8b-e6e7-46f5-a131-aad57f81fa2b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:55.549443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b2eb8b-e6e7-46f5-a131-aad57f81fa2b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:55.551323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b2eb8b-e6e7-46f5-a131-aad57f81fa2b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:55.573874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 285/557] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:57:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:57:55.576142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a4d8f-79dc-40f6-aacc-ad3ec8fbc321 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.576606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a4d8f-79dc-40f6-aacc-ad3ec8fbc321 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.577032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308a4d8f-79dc-40f6-aacc-ad3ec8fbc321 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.579093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 286/558] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:55.579946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308a4d8f-79dc-40f6-aacc-ad3ec8fbc321 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.649460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 273/559] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:57:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:57:55.655710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f88d293-91cb-4ffc-823e-3ac1c3a6b651 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.656010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f88d293-91cb-4ffc-823e-3ac1c3a6b651 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.656280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f88d293-91cb-4ffc-823e-3ac1c3a6b651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.672903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2f88d293-91cb-4ffc-823e-3ac1c3a6b651 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:55.674011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2f88d293-91cb-4ffc-823e-3ac1c3a6b651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:55.690891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308a4d8f-79dc-40f6-aacc-ad3ec8fbc321 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:55.693257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:55.729162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4031e47-0a57-4d64-a0c2-d5007234d310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:55.729956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4031e47-0a57-4d64-a0c2-d5007234d310 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.730927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4031e47-0a57-4d64-a0c2-d5007234d310 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.731166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4031e47-0a57-4d64-a0c2-d5007234d310 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.741065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 274/560] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:55.742383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8704f89-0d5b-4951-afe7-061111577352 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.742671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8704f89-0d5b-4951-afe7-061111577352 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.742947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8704f89-0d5b-4951-afe7-061111577352 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.744353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 275/561] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:55.744910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8704f89-0d5b-4951-afe7-061111577352 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.753631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4031e47-0a57-4d64-a0c2-d5007234d310 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:55.755312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4031e47-0a57-4d64-a0c2-d5007234d310 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:55.772138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b4031e47-0a57-4d64-a0c2-d5007234d310 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:55.776264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:55.777763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:55.778657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:55.778657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:55.778657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:55.778657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:55.780374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 287/562] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:55.781827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c27017-fddf-4063-be90-9b374b764715 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.782124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c27017-fddf-4063-be90-9b374b764715 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.782405 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c27017-fddf-4063-be90-9b374b764715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.800009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c2c27017-fddf-4063-be90-9b374b764715 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:55.801243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c2c27017-fddf-4063-be90-9b374b764715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:55.862906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8704f89-0d5b-4951-afe7-061111577352 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:55.874894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 288/563] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:55.875866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c27017-fddf-4063-be90-9b374b764715 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.910564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:55.911353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:55.911633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:55.911899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:55.928308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:55.929794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:55.948431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:55.949540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:55.950439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:55.950439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:55.950439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:55.950439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:55.950439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 276/564] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:55.950756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:55.976243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:55.994411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c27017-fddf-4063-be90-9b374b764715 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.028919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.038521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.039356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.039647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.039983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.052411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:56.061176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:56.062095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:56.062962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:56.063602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0d6ec5-f878-40b8-acc2-b6d20ad4c578 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.063676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 289/565] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:56.063676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.102773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.103441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.103726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.104031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.104797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:56.105109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.106492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.118177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 277/566] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:56.118960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.167069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6db39d-6bfe-4a5b-8851-7670ecf60bac None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.200882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.201680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.201966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.202244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.213328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:56.221090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:56.222171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 290/567] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:56.223031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.231986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.232330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0334883-10d9-4a88-987c-49c3899ab6dc None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.260250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.266720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77092824-035f-41d2-bd98-3c43becfce90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.267634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77092824-035f-41d2-bd98-3c43becfce90 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.267914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77092824-035f-41d2-bd98-3c43becfce90 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.268190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77092824-035f-41d2-bd98-3c43becfce90 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.268936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77092824-035f-41d2-bd98-3c43becfce90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:56.269242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77092824-035f-41d2-bd98-3c43becfce90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.270607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77092824-035f-41d2-bd98-3c43becfce90 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.282884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 278/568] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:56.284220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0932036-13ec-4653-9098-b7abefb8b2f3 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.284554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0932036-13ec-4653-9098-b7abefb8b2f3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.284832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0932036-13ec-4653-9098-b7abefb8b2f3 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.286051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 279/569] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:56.286742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0932036-13ec-4653-9098-b7abefb8b2f3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.328882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ef1ea5-bdc1-4966-af81-58bec4bbff8f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.360587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.361251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.361532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.361809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.362540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:56.362843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.364207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.372413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 291/570] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:56.373201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.390015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0932036-13ec-4653-9098-b7abefb8b2f3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.425082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.425868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.426146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.426417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.434522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.435971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.437348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:56.451174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:56.452217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:56.453310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 280/571] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:56.454217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.472635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66068ea-d50e-4b4e-ac66-c767744a0f8a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.504513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.505410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc637dc-dbeb-471c-be2c-6fa24baa6621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.506017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc637dc-dbeb-471c-be2c-6fa24baa6621 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.506372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc637dc-dbeb-471c-be2c-6fa24baa6621 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.506686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc637dc-dbeb-471c-be2c-6fa24baa6621 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.507412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc637dc-dbeb-471c-be2c-6fa24baa6621 None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:56.507931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc637dc-dbeb-471c-be2c-6fa24baa6621 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.509101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc637dc-dbeb-471c-be2c-6fa24baa6621 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.520484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 292/572] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:56.520484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce155a1-1cad-43a0-b160-06435e2b2ec5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.520484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce155a1-1cad-43a0-b160-06435e2b2ec5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.520484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce155a1-1cad-43a0-b160-06435e2b2ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.536201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-2ce155a1-1cad-43a0-b160-06435e2b2ec5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:56.537417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2ce155a1-1cad-43a0-b160-06435e2b2ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:56.561314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6942aa-ed95-40fa-82a7-3c6f43a3181a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.596249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.597073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.597360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.597639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.601446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 293/573] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:56.602261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce155a1-1cad-43a0-b160-06435e2b2ec5 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.604356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.605873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:56.621427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:56.622309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:56.623171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:56.627058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 281/574] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:56.627869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.663115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.702751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce155a1-1cad-43a0-b160-06435e2b2ec5 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.726875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f42847b-654e-443c-b878-5c524f2c5a5a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.736011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.736833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.737121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.737394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.738660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.747878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.749341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.762847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.763498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.763781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.764056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.764868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:56.765178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:56.766447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:56.767597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:56.768508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:56.768508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:56.768508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:56.768508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:56.768721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.768803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 294/575] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:56.768803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.779589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 282/576] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:56.780412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.875944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6828367b-e4ad-4453-a8da-ef2471675c37 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.877459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c10b25-4579-45cf-8e86-e9ceed2993c0 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:56.888262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:56.910681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.911652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.911776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:56.911941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.912219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.912600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:56.912887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:56.913163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:56.925117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:57:56.926548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:56.928018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:57:56.932993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:56.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:56.935138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 283/577] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:56.935942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:56.947162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:56.948058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:56.949026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:56.949026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:56.949026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:56.949026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:56.949026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 295/578] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:56.949308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:56.971661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.041443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf90a0b-1be6-403c-b386-9151a73ad1fb None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.046923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10bf7fe-a2c1-4cab-8a22-5884d955e018 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.074103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.074760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.075045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.075323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.076112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:57.076442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.077875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.079109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.079753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.080038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.080326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.081065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:57.081369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.082707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.090182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 284/579] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:57.090944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.094948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 296/580] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:57.095703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.122654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.181720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.191243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd4b0f-2c87-4df3-ba14-8ffa341a486c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.195889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aac5ca-d4da-49fc-a3df-f5811ec589ae None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.224465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.225295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.225577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.225849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/f99a9fe4b40646d389c3ebf47358956c/roles/0d7a50b0cac3467b8b2000f706f52329` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.229628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.230930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.231220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.231497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.232242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:57.232582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.233945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.240684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=f99a9fe4b40646d389c3ebf47358956c, role_id=0d7a50b0cac3467b8b2000f706f52329)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.241830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 297/581] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:57.242119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.242616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.255233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 285/582] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:57 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/f99a9fe4b40646d389c3ebf47358956c/roles/0d7a50b0cac3467b8b2000f706f52329 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:57.256414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.353374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.372923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74926cd-8ffa-42dd-a17f-3f8f4f8e7e46 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.373180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3dce9b-1bdb-468a-846a-de5387492815 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.413201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.413869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.414156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.414440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.415285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=c1d16fdc498e4d84bf8657c7f55d6b26) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:57.415600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.417099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.418773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.419448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.419739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.420018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.420951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:57.421267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.422640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.423820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 286/583] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/role_assignments?user.id=c1d16fdc498e4d84bf8657c7f55d6b26&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:57.424738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.429289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.443007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 298/584] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:57.443969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.568703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06905012-d035-49f8-ad90-dd74d075c025 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.575855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c85382-caba-404b-95e3-3706eb9a4bb3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.609607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.610418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.610706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.610981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.617800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.618978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.619469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.619944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.623214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.625327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.629888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.631746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.632913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:57.647754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:57.648855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:57.649736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:57.649736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:57.649736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:57.649736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:57.654478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 299/585] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:57.654478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:57.657152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:57.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 05 09:57:57.659045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:57:57.661671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 287/586] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/users/glance => generated 84 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:57.662667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.733420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.776479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5f8ba1-1bfb-4b1d-9a49-6cb663ff0cc2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.794427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc8fee6-9960-45d7-9057-c1a07b321243 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.816917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.817648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.817936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.818217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.819050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:57.819309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.820717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.836305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:57.841578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 288/587] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:57.841578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.841767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:57.841767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:57.841767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:57.853689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:57.855844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:57.874261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:57.875236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:57:57.876664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:57.880186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 300/588] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/users/cinder => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:57.881078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:57.966191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1272c8cc-cde2-4b53-a70d-2ae215c16e20 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:57.966684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:57.992636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.005245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.006235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.006746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.007228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.008272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:58.008811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.010438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.012524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b41619d-3ed7-4f6d-b816-cc7eb95c6f2b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.030713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 289/589] 10.222.0.20 () {56 vars in 1137 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:58.031836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.049076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.049937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.050219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.050499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.058736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.060208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:57:58.075525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:58.076491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 05 09:57:58.077453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:58.080681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 301/590] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:57 2022] GET /identity/v3/users/neutron => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:58.081475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.144743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13448f5-536f-4b46-a716-254ff9f33703 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.182156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.183231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.184252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.184758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.185804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:58.186316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.187911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f605a874-827b-4174-b77b-6e9dbffbebbb None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.188194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.205941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.212142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 290/591] 10.222.0.20 () {56 vars in 1137 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:58.213418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.232060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.232915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.233204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.233483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.244885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.246743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.260563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:58.264506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:58.265396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:58.266286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:58.266286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:58.266286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:58.266286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:58.266286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 302/592] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:58.266972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.324590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994ed1ce-543d-49d4-b006-7c8d890a15e8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.361612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.362629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.363145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.363636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.364732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:58.365246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.366892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.370190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9687ba23-2f27-4405-91de-465fb265c785 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.386237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 291/593] 10.222.0.20 () {56 vars in 1139 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:58.387513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.411353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.412223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.412463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.412732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.424142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.425649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:58.443229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:58.444165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:58.445107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:58.445107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:58.445107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:58.445107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:58.449136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 303/594] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:58.449888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.476823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.495074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.499157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38652262-6154-4cf1-8e4a-287bd48483b1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.535284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.536211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.536713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.537174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.538110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:58.538608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.540156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.553052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 292/595] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:58.554188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.560325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83908bd4-cc81-4b43-a4bb-840c66e07605 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.598754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.599643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.600007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.600688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.611352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.613091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:57:58.634500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:58.635441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:58.636364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:57:58.636364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:57:58.636364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:57:58.636364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:58.640936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 304/596] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:58.641853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.666481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75135d32-5778-4b3c-8972-cb8e9b80ead4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.702628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.703553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.704051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.704642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.705649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:58.706145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.707795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.720862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 293/597] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:58.721889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.726930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.744815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.756655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb7969c-a033-409b-bcd8-d139f492c24c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.794188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.795143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.795501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.795848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.803194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.804736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:58.819931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:58.821039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:58.821974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 305/598] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:58.822472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.838297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ff14a8-50b8-409e-a6df-754b07d0e9a4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.876818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.877811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.878304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.878764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.879668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:58.880192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.881821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:58.895630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 294/599] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:58.896804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:58.945081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f325a4bf-a91d-4870-8038-b82a88c48d36 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:58.983008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:58.983647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:58.983937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:58.984218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:58.986378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.988741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:58.993018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:58.994810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:59.010757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:59.012275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:59.013534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:59.015606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72ba39d-81e9-449f-a2a4-186d4fbc709e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.017646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 306/600] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:59.018526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.052675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.053717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.054208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.054733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.055755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:59.056269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:59.057965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.071189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 295/601] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:59.072409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.127325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67962a5-55ab-4fcc-9784-bf220d1fb802 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.164346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.165561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.165854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.166137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.176198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:59.178235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.181854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1464ee21-15fd-44bf-bd2f-ae7f3d9db115 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:57:59.197430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:57:59.198348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:57:59.199281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:57:59.203541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 307/602] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:57:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:57:59.203541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.225114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.225892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.226180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.226454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.227300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:59.227606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:59.229327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.237900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:59.248532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 296/603] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:59.249537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.277952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:59.336594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3869e4bf-f3a8-4d1e-9449-077696316319 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.365541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf88015e-1253-47af-b92f-01c8e1441247 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.375384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.376770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.377062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.377396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/90d9b9421d454465b8f99137cd182abf/roles/0d7a50b0cac3467b8b2000f706f52329` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.393193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=90d9b9421d454465b8f99137cd182abf, role_id=0d7a50b0cac3467b8b2000f706f52329)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:59.394832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.399709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8a9e63-a678-42c2-a904-24013104c01e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.400493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a9e63-a678-42c2-a904-24013104c01e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.400821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a9e63-a678-42c2-a904-24013104c01e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.401100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8a9e63-a678-42c2-a904-24013104c01e None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.401880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8a9e63-a678-42c2-a904-24013104c01e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:57:59.402839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8a9e63-a678-42c2-a904-24013104c01e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:59.404223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8a9e63-a678-42c2-a904-24013104c01e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.408133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 308/604] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:59 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/90d9b9421d454465b8f99137cd182abf/roles/0d7a50b0cac3467b8b2000f706f52329 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:59.409147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.428454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 297/605] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:57:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:57:59.428454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf07c64-2b3d-43c0-88df-6bc2548faecf None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.428454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf07c64-2b3d-43c0-88df-6bc2548faecf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.428454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf07c64-2b3d-43c0-88df-6bc2548faecf None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.431552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 298/606] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:59 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:59.432541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf07c64-2b3d-43c0-88df-6bc2548faecf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.509338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:59.563069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62398fd0-3ee4-40c0-a845-9602ab05bf98 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.566070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:59.568274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf07c64-2b3d-43c0-88df-6bc2548faecf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.606856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c61f6e-f22c-42e1-a67e-20048c380b51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.607755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c61f6e-f22c-42e1-a67e-20048c380b51 None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.608044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c61f6e-f22c-42e1-a67e-20048c380b51 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.608355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c61f6e-f22c-42e1-a67e-20048c380b51 None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/1fc5b3da67de48cb83848168b16dd153/roles/0d7a50b0cac3467b8b2000f706f52329` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.609584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ea750b-885e-48de-95ad-26c3325547a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.610706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ea750b-885e-48de-95ad-26c3325547a6 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.610943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ea750b-885e-48de-95ad-26c3325547a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.611284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ea750b-885e-48de-95ad-26c3325547a6 None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/36ce9bd8ce6448b8b09fe9eb7b8f7c54/roles/0d7a50b0cac3467b8b2000f706f52329` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.624496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c61f6e-f22c-42e1-a67e-20048c380b51 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=1fc5b3da67de48cb83848168b16dd153, role_id=0d7a50b0cac3467b8b2000f706f52329)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:59.626019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c61f6e-f22c-42e1-a67e-20048c380b51 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.633435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ea750b-885e-48de-95ad-26c3325547a6 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=36ce9bd8ce6448b8b09fe9eb7b8f7c54, role_id=0d7a50b0cac3467b8b2000f706f52329)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:57:59.634813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ea750b-885e-48de-95ad-26c3325547a6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:57:59.641314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 309/607] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:59 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/1fc5b3da67de48cb83848168b16dd153/roles/0d7a50b0cac3467b8b2000f706f52329 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:59.644740 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252aa9fa-e709-4fbb-b997-6a4879f279f5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.644740 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252aa9fa-e709-4fbb-b997-6a4879f279f5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.644740 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252aa9fa-e709-4fbb-b997-6a4879f279f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.648632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 299/608] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:57:59 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/36ce9bd8ce6448b8b09fe9eb7b8f7c54/roles/0d7a50b0cac3467b8b2000f706f52329 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:57:59.657702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddbcfc4-5a21-429a-8975-8f1a5f0d259a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.658046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddbcfc4-5a21-429a-8975-8f1a5f0d259a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.658333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddbcfc4-5a21-429a-8975-8f1a5f0d259a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.660229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 300/609] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:57:59 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:57:59.666624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef4bc1d-4529-41ac-9f77-045fd3afc4a7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.666912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef4bc1d-4529-41ac-9f77-045fd3afc4a7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.667313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef4bc1d-4529-41ac-9f77-045fd3afc4a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:57:59.668947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-252aa9fa-e709-4fbb-b997-6a4879f279f5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:59.670376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-252aa9fa-e709-4fbb-b997-6a4879f279f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:59.701596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8ef4bc1d-4529-41ac-9f77-045fd3afc4a7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:57:59.702844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8ef4bc1d-4529-41ac-9f77-045fd3afc4a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:57:59.752991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 310/610] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:57:59.779318 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252aa9fa-e709-4fbb-b997-6a4879f279f5 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.817666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 301/611] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:57:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:57:59.854164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef4bc1d-4529-41ac-9f77-045fd3afc4a7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:57:59.904606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:59.928609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252aa9fa-e709-4fbb-b997-6a4879f279f5 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:57:59.943346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:57:59.988243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:57:59.989100 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:57:59.989389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:57:59.989671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.003911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:00.014363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:00.015330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:00.016169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:00.018346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 311/612] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:00.022848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.030242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef4bc1d-4529-41ac-9f77-045fd3afc4a7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.077842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:00.079272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:00.079790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:00.080743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.095711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:00.106967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:00.107925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:00.111541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 302/613] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:57:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:00.118554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.207286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b420d94a-bcc3-4397-b9dd-7f2f6e45d39c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.242333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:00.247482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95afc0af-7429-4d3d-b553-c42efcfd1435 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.285935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:00.286967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:00.287457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:00.287962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.288974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:00.289487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:00.291250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:00.300968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 303/614] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:00.309511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.322944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:00.323618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:00.324518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:00.324809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.325624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:00.325930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:00.327382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:00.348417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 312/615] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:00.349574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.368754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:00.491925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1e4003-8d99-4641-bb21-2478d56b8d4e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.512187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e51d4f9-0586-43c9-9b99-b277e48bc418 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.527679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:00.528578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:00.528869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:00.529146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.540734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:00.542526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:00.560865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:00.561790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:00.562660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:00.562660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:00.562660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:00.562660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:00.564521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:00.565557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:00.566041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:00.566534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.566705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 313/616] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:00.567890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.585488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:00.592420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:00.620174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:00.621743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:00.622626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:00.622626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:00.622626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:00.622626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:00.624973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 304/617] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:00.635778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:00.638450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.668447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:00.684809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d357ef10-2106-428a-985e-683a436dc2f1 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.720088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:00.720805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:00.721093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:00.721372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.722097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:00.722407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:00.723778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:00.740510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 314/618] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:00.741880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.840258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270478d0-6e32-4114-9f03-70d254ab007c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.903250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:00.905435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:00.908550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:00.909061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:00.909984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:00.910449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:00.912114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:00.943496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 305/619] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:00.945384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398a8e4-a0eb-4c5f-af78-f1096d73a98c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:00.948339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:00.979839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:01.014041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.014838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.015124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.015399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.025860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.027843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:01.054705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:01.055843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:01.056826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:01.057368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 315/620] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/users/nova => generated 82 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:01.063266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:01.078725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:01.126662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19726405-c478-404b-9c21-5a5e95c99a1b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:01.202223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.202464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.202556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.203455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] PATH_INFO: `/v3/users/swift` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.213024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.215277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:01.217767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba697a3f-6a7b-4143-8cb8-edf6682e1192 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:01.235789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:01.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:01.237789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 306/621] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:01.249631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:01.276244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.277329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.277823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.278323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.279409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:01.281470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.283133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:01.326730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 316/622] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:58:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:01.337393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:01.377340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:01.406367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4c6510-d2d5-4300-b9ed-fdcf7dcc9aca None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:01.436807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:01.442296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.443237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.443237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.443571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.444377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:01.444692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.446147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:01.450109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c53582-b33e-4e36-bda0-fc62e48c3669 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:01.466165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 307/623] 10.222.0.20 () {56 vars in 1135 bytes} [Tue Jul 5 09:58:01 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:01.471573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:01.523390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.524056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.524340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.524640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.588585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0da834e-3890-49ea-a355-7e2f5f240703 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:01.624920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.625723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.626011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.626290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.637545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.638932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:01.658194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:01.659143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:01.660030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:01.660030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:01.660030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:01.660030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:01.660265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.660265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:01.660415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 308/624] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:01.669498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:01.702480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:01.780578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 317/625] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:58:01.784092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:01.794384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:01.853492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f87b26-d7e7-4f48-aac9-230885b443fd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:01.894730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3203057c-602e-4328-944a-0874432590f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.895727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3203057c-602e-4328-944a-0874432590f9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.896166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3203057c-602e-4328-944a-0874432590f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.896603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3203057c-602e-4328-944a-0874432590f9 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.897342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3203057c-602e-4328-944a-0874432590f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:01.897914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3203057c-602e-4328-944a-0874432590f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.900095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3203057c-602e-4328-944a-0874432590f9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:01.917300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4037ae9-287e-4f03-b43a-9fb9c7697733 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:01.920795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 309/626] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:01.921876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d37f954-956d-4629-a5e0-0a8d63c1bae1 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.922315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d37f954-956d-4629-a5e0-0a8d63c1bae1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.922603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d37f954-956d-4629-a5e0-0a8d63c1bae1 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.925084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 310/627] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:01.932297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d37f954-956d-4629-a5e0-0a8d63c1bae1 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:01.977593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b6fb13-30b6-49d4-aa16-253bcd544358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:01.978404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6fb13-30b6-49d4-aa16-253bcd544358 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:01.978687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6fb13-30b6-49d4-aa16-253bcd544358 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:01.979105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b6fb13-30b6-49d4-aa16-253bcd544358 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:01.982846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:01.995278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b6fb13-30b6-49d4-aa16-253bcd544358 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:01.997084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b6fb13-30b6-49d4-aa16-253bcd544358 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-a4b6fb13-30b6-49d4-aa16-253bcd544358 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:02.021343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:02.023490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 318/628] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05063600-752b-405e-8348-17c9ae1b7647 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05063600-752b-405e-8348-17c9ae1b7647 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.025092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05063600-752b-405e-8348-17c9ae1b7647 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.048213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-05063600-752b-405e-8348-17c9ae1b7647 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:02.050133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-05063600-752b-405e-8348-17c9ae1b7647 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:02.052617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d37f954-956d-4629-a5e0-0a8d63c1bae1 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.054227 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:02.091324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.092132 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.092441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.092725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.099949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:02.101441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:02.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:02.116099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:02.117005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 311/629] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:02.118039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.125921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 319/630] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:02.126842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011a035-3323-4b1f-80ff-4d894b0a2fce None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.127129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011a035-3323-4b1f-80ff-4d894b0a2fce None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.127404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a011a035-3323-4b1f-80ff-4d894b0a2fce None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.128885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 320/631] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:02.133372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a011a035-3323-4b1f-80ff-4d894b0a2fce None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.235894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d5af04-dd6e-4cb4-9770-092c651e0cdf None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.242769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a011a035-3323-4b1f-80ff-4d894b0a2fce None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.260533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:02.271552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c6caaf-653a-4141-86b9-27fe661cf87c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.272218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c6caaf-653a-4141-86b9-27fe661cf87c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.272545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c6caaf-653a-4141-86b9-27fe661cf87c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.272824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c6caaf-653a-4141-86b9-27fe661cf87c None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.273544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c6caaf-653a-4141-86b9-27fe661cf87c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:02.273858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c6caaf-653a-4141-86b9-27fe661cf87c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:02.275232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c6caaf-653a-4141-86b9-27fe661cf87c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.278515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.279166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.279448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.279721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.280535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:02.280841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:02.282238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.286939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 312/632] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:02.288130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30096524-fa03-4927-b149-8bef81b31342 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.288458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30096524-fa03-4927-b149-8bef81b31342 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.288739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30096524-fa03-4927-b149-8bef81b31342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.298255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 321/633] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:02.298255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.304587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-30096524-fa03-4927-b149-8bef81b31342 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:02.305781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-30096524-fa03-4927-b149-8bef81b31342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:02.323033 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:02.373650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 313/634] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:02.374942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3674a2-0c4a-49de-8f3f-1f3fb4417400 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.375231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3674a2-0c4a-49de-8f3f-1f3fb4417400 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.375507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3674a2-0c4a-49de-8f3f-1f3fb4417400 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.376818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 314/635] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:02.377510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3674a2-0c4a-49de-8f3f-1f3fb4417400 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.396466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb19148-fa30-43b7-a2a4-45faa700a135 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.428963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.429753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.430036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.430311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.441335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:02.449558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:02.450651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 322/636] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:02.451647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.485685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3674a2-0c4a-49de-8f3f-1f3fb4417400 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.489608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:02.520083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2720c53-8996-42b6-bc99-473f734ac94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.520914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2720c53-8996-42b6-bc99-473f734ac94b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.521191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2720c53-8996-42b6-bc99-473f734ac94b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.521464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2720c53-8996-42b6-bc99-473f734ac94b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.528682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2720c53-8996-42b6-bc99-473f734ac94b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:02.530130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2720c53-8996-42b6-bc99-473f734ac94b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a2720c53-8996-42b6-bc99-473f734ac94b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:02.543912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:02.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:02.545704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 315/637] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:02.546278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e887a-3fab-465d-a2a8-e57a91d10bef None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.546570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e887a-3fab-465d-a2a8-e57a91d10bef None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.546874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e887a-3fab-465d-a2a8-e57a91d10bef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.551643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a16ec6f-7dbd-4933-8a60-dd52e524f27b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.563248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-329e887a-3fab-465d-a2a8-e57a91d10bef None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:02.564475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-329e887a-3fab-465d-a2a8-e57a91d10bef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:02.571747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:02.585630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.586519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.586804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.587076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/c1d16fdc498e4d84bf8657c7f55d6b26/roles/0d7a50b0cac3467b8b2000f706f52329` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.601450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=c1d16fdc498e4d84bf8657c7f55d6b26, role_id=0d7a50b0cac3467b8b2000f706f52329)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:02.602896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.614803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 323/638] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:58:02 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/c1d16fdc498e4d84bf8657c7f55d6b26/roles/0d7a50b0cac3467b8b2000f706f52329 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:58:02.615591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.634533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 316/639] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:02.635436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329e887a-3fab-465d-a2a8-e57a91d10bef None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.726816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:02.747558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a42e2d-b6e4-44e3-b8c3-bcefb3b2e47f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.770638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329e887a-3fab-465d-a2a8-e57a91d10bef None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.794103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.795002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.795358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.795713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.807420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:02.809439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.810114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.810400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.810680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.811435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:02.811748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:02.813158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:02.815746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:02.816703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:02.817511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 324/640] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:02.818442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.822684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 317/641] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:02.827663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.831434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:02.928773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad08cf5b-86b8-46e1-bcce-df02703a66d8 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.941288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc0406-a5b7-4451-8424-b4de47f8d677 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:02.965577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.966343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.966694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.966975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.967765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:02.968070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:02.969540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:02.976967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:02.977767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:02.978055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:02.978335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:02.987245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 325/642] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:02.987245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:02.993919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:02.996658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:03.002650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:03.003650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:03.004562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 318/643] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:03.005256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.077618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.102335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcf0485-38af-42f4-be2b-e28e76d7bdc5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.119755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35bf9ea-0bd0-4a6c-95e6-eb0491055137 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.141033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.142038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.142038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.142253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.143250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:03.143557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.144974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.153644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 326/644] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:03.154446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.157783 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.158819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.159288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.159564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.170842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.172297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:03.190030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:03.190938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:03.191991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:03.191991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:03.191991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:03.191991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:03.191991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 319/645] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:03.191991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.235432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.269030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502bc1e6-fe97-4ee0-aaf3-bc9a18569ba4 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.299486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f798f7bf-f0b4-479c-b672-b5502322eabc None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.304357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.305054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.305343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.305625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.306466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=f99a9fe4b40646d389c3ebf47358956c) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:03.306780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.308332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.314533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 327/646] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/role_assignments?user.id=f99a9fe4b40646d389c3ebf47358956c&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:03.315327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.327444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.349790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.350746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.351215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.351748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.352855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:03.353322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.355100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.365453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 320/647] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:03.366622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.437999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99493987-a9b6-4887-9bdd-37907cf45cce None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.489533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.490954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.491415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.491843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.495246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.504311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10aa26c-64ba-4045-96e0-a33143174aef None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.510186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.511701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:03.536337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:03.537276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:03.538202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:03.538202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:03.538202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:03.538202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:03.538202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 328/648] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:03.538202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.559087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.559767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.560087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.560373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.561169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:03.561480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.562933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.576368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 321/649] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:03.577073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.627872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.672067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b51f8-12d2-4b85-b8e4-1980856de0d7 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.705417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ed9465-cca4-4dbd-9846-6567ce4f31fe None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.711635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.712726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.713231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.713698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.725856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.727722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.748365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:03.748966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:03.749810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:03.750703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:03.750703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:03.750703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:03.750703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:03.750918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.750918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.750918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.750918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:03.750918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.752041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.758045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 329/650] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:03.759189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.769951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 322/651] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:03.769951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.806159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.871397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56755a-8f34-40b4-b2ab-bdd730f400e8 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.895735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:03.900296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88309ef-fbd2-46e2-9c51-66ce2a249c16 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:03.910258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.911336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.911797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.912245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] PATH_INFO: `/v3/users/glance` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.928587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.931257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.940905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:03.941584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:03.942104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:03.942267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:03.942995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:03.943302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:03.944706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:03.951314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:03.952224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:03.953236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 330/652] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/users/glance => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:03.953899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:03.957679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 323/653] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:03.960079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.064291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:04.068109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce320cbf-b61b-41a7-b362-5bcd0357bb7d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.099950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c01d23-dc50-4f25-9459-bc1de0d49c53 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.104550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.105024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.105807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.106083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.114052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.115614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:04.131969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:04.133205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:58:04.134138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:04.138705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 331/654] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/users/cinder => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:04.139530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.147357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.148304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.149311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.149808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.151373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:04.151901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.154860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.181131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 324/655] 10.222.0.20 () {56 vars in 1137 bytes} [Tue Jul 5 09:58:03 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:04.182367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.202806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:04.266586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9e4337-25de-4850-8e96-03966cfc568d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.289593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cc9120-0c92-40bc-a0e9-9130b65f656e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.308613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0139b316-f404-4882-a743-c523e69de43a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.309408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0139b316-f404-4882-a743-c523e69de43a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.309694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0139b316-f404-4882-a743-c523e69de43a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.309972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0139b316-f404-4882-a743-c523e69de43a None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.317840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0139b316-f404-4882-a743-c523e69de43a None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.319323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0139b316-f404-4882-a743-c523e69de43a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.332527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.333461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.333948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.334420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-0139b316-f404-4882-a743-c523e69de43a None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:04.335386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:04.336276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 332/656] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:04 2022] GET /identity/v3/users/neutron => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:04.337621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0139b316-f404-4882-a743-c523e69de43a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.338707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:04.339224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.341454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.363689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:04.365914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 325/657] 10.222.0.20 () {56 vars in 1137 bytes} [Tue Jul 5 09:58:04 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:04.368116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.456680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0139b316-f404-4882-a743-c523e69de43a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.470459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:04.496476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.497434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.497893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.498337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.503467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae7bb85-b6c0-4d72-a419-2a982c8ffbf5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.549633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.550424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.550900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.551387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.552322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:04.552744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.555496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.575179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.576998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.580505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 326/658] 10.222.0.20 () {56 vars in 1139 bytes} [Tue Jul 5 09:58:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:04.580505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.601990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:04.697191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 333/659] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:58:04.698162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.713349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8284263-80b1-4db6-9621-5bd4f1b74410 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.745756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:04.751368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dec982-5e9a-4bfe-a9d5-eb872c465229 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.752767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dec982-5e9a-4bfe-a9d5-eb872c465229 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.753096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dec982-5e9a-4bfe-a9d5-eb872c465229 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.753703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dec982-5e9a-4bfe-a9d5-eb872c465229 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.809289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54d164d-01fb-45db-9238-14477affaddd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:04.826709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:04.827543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dec982-5e9a-4bfe-a9d5-eb872c465229 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.829141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0dec982-5e9a-4bfe-a9d5-eb872c465229 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.846162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:04.846817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.847106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.847406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.916456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:04.917958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:04.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 327/660] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:58:04.941058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4a88a-fecb-4f3f-8175-8add73372394 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:04.941058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4a88a-fecb-4f3f-8175-8add73372394 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:04.941058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e4a88a-fecb-4f3f-8175-8add73372394 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:04.941058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 328/661] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:04.941058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e4a88a-fecb-4f3f-8175-8add73372394 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:04.978518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.028725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 334/662] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:58:05.029581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.049538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.059165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e4a88a-fecb-4f3f-8175-8add73372394 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.096250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ad1bb5-b7a4-4894-a610-aa4916dd4531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.097090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad1bb5-b7a4-4894-a610-aa4916dd4531 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.097375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad1bb5-b7a4-4894-a610-aa4916dd4531 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.097646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad1bb5-b7a4-4894-a610-aa4916dd4531 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.108279 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad1bb5-b7a4-4894-a610-aa4916dd4531 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.109866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad1bb5-b7a4-4894-a610-aa4916dd4531 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-d4ad1bb5-b7a4-4894-a610-aa4916dd4531 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:05.128573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:05.129555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:05.130591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:05.130591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:05.130591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:05.130591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:05.137091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9e31a3-b900-4bae-ab2b-1740e1384990 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 329/663] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:05.139345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48966962-87aa-42a3-8225-9383a4522656 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.139802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48966962-87aa-42a3-8225-9383a4522656 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.140247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48966962-87aa-42a3-8225-9383a4522656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.157514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-48966962-87aa-42a3-8225-9383a4522656 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:05.159034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-48966962-87aa-42a3-8225-9383a4522656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:05.172223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.173065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.173346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.173622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.184420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.185864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:05.204826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:05.205755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:05.206699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:05.206699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:05.206699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:05.206699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:05.206699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 335/664] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:05.206976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.234689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 330/665] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:05.237264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48966962-87aa-42a3-8225-9383a4522656 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.258136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.287013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.321157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d132356-a609-447e-ae47-2b21c3d2414d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.347351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48966962-87aa-42a3-8225-9383a4522656 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.357734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.358862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.359147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.359426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.370579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.372802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.382994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.383656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.383936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.384212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.384961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:05.385267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.387229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:05.391414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:05.392300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:05.398622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:05.398622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:05.398622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:05.398622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:05.398622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 336/666] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:05.398622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.403644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 331/667] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:05.408137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.487176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.509127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29b4583-c691-4cac-81b0-654d55f65549 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.513789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17bf01b-c8d8-4bd2-bc33-d153ee1a521a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.533216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.546956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.547580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.547873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.548158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.548918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:05.549230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.550618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.550865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.551703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.551995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.552266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.563343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 337/668] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:05.563526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:05.564152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:05.571005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:05.572125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 332/669] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:05.573389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.665712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc606b1-1021-4af7-828d-e3b715933acd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.672469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2debd-5f04-4658-bb2e-af63f2473cc3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.699526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cb2a8c-6b54-4d81-9e10-25bf9dd5993e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.700181 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb2a8c-6b54-4d81-9e10-25bf9dd5993e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.700494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb2a8c-6b54-4d81-9e10-25bf9dd5993e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.700783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cb2a8c-6b54-4d81-9e10-25bf9dd5993e None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.701502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb2a8c-6b54-4d81-9e10-25bf9dd5993e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:05.701810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb2a8c-6b54-4d81-9e10-25bf9dd5993e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.703177 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cb2a8c-6b54-4d81-9e10-25bf9dd5993e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.705389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.706174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.706458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.706759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.708686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.714123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.714766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 338/670] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:05.715808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.716019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283ef39a-2d35-4e8f-a8ba-a76720bdee74 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.716307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283ef39a-2d35-4e8f-a8ba-a76720bdee74 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.716616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283ef39a-2d35-4e8f-a8ba-a76720bdee74 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.717857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 339/671] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:05.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283ef39a-2d35-4e8f-a8ba-a76720bdee74 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:05.735585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:05.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:05.737459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 333/672] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:05.737983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.766381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.822957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283ef39a-2d35-4e8f-a8ba-a76720bdee74 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.837342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28537a3-2ae9-4a99-bda1-03ad486b1d61 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.856468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.857258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.857552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.857828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.864383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.865861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.871492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d7a96a-a044-4773-8ff3-c16bf5ac1751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:05.872161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7a96a-a044-4773-8ff3-c16bf5ac1751 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.872492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7a96a-a044-4773-8ff3-c16bf5ac1751 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.872760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d7a96a-a044-4773-8ff3-c16bf5ac1751 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.873507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d7a96a-a044-4773-8ff3-c16bf5ac1751 None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:05.873815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d7a96a-a044-4773-8ff3-c16bf5ac1751 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:05.875176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d7a96a-a044-4773-8ff3-c16bf5ac1751 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:05.880004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:05.880894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:05.881773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 340/673] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:05.882668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.883058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 334/674] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:05.884272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fea1d4-a732-4182-b06e-3bb3471c280d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:05.884595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fea1d4-a732-4182-b06e-3bb3471c280d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:05.884878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fea1d4-a732-4182-b06e-3bb3471c280d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:05.901046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d5fea1d4-a732-4182-b06e-3bb3471c280d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:05.902771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d5fea1d4-a732-4182-b06e-3bb3471c280d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:05.952562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:05.969607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 335/675] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:05.970216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fea1d4-a732-4182-b06e-3bb3471c280d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:05.991280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba61025-2258-4551-801e-bc1e6ea81588 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:05.992903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.026786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.028213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.028531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.028812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.035366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.036851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:06.051373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:06.052240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:06.053159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 341/676] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:06.054022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.072706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fea1d4-a732-4182-b06e-3bb3471c280d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.107461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.108124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.108430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.108735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.109508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.109823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.111258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.122811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 336/677] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.123607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.153947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87539778-d87f-4b65-875e-6c116d8f667d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.170067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.188026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.188719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.189009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.189287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.190074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.190389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.191810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.203017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 342/678] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.203814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.226704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1526f507-6e83-4db9-ab00-fc8be65d7254 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.228195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.260602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.261397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.262211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.262491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.273321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.274774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:06.292642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:06.293563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:06.294681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:06.294681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:06.294681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:06.294681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:06.294681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 337/679] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:06.294681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.313698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceff8a8c-2e0b-4fbe-9ff1-f9442ccd3e03 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.347720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.348551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.348841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.349118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.359079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:06.366684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:06.367759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:06.368677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 343/680] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:06.369398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.390965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.395342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13244b6-d4eb-4cc9-a1d5-f3c39a956473 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.429527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.430198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.430498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.430768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.431557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.431867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.433347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.445183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 338/681] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.445918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.457335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.476233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3878745-914a-4583-8e8d-9b0a3d91e60d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.508357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.509107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.509472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.509886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.510784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=90d9b9421d454465b8f99137cd182abf) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.511097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.513106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.519018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 344/682] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/role_assignments?user.id=90d9b9421d454465b8f99137cd182abf&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.519815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.548924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e7ca2-40a3-4b69-ae05-252b2bc25e49 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.590188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.591021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.591399 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.591881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.592992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=1fc5b3da67de48cb83848168b16dd153) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.593405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.594951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.604942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 339/683] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/role_assignments?user.id=1fc5b3da67de48cb83848168b16dd153&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.606895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.638153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06590377-f4ab-4c2e-92c9-bf8d18f0ad9c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.643761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.691131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.691828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.693830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.693830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.693830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.693830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.694884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.717997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 345/684] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.717997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.733962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.748235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625ae000-66fe-481d-b8a0-7bfd5b4b6a6e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.793590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.794640 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.795127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.795647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.796560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.796985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.798471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.808902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 340/685] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.809658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.858577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc830c24-34aa-45e0-91df-ad0b4449faa0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:06.900991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:06.901966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:06.902945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:06.903325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:06.904564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=36ce9bd8ce6448b8b09fe9eb7b8f7c54) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:06.904985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:06.907431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:06.915394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 346/686] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/role_assignments?user.id=36ce9bd8ce6448b8b09fe9eb7b8f7c54&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 608 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:06.920243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:06.939312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:06.980231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e336011f-e66f-41ef-9383-1d8e03b697e6 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:07.059846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:07.061028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:07.062221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:07.062710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:07.065440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:07.084692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:07.088300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:07.116014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:07.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:07.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:07.119714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 341/687] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:07.126676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:07.128770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae434dca-7d87-4116-ba74-18fefb64c45f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:07.184468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:07.186073 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:07.186559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:07.187053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:07.199757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:07.201598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:07.223209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:07.224120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:07.225037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:07.225037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:07.225037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:07.225037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:07.227291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 347/688] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:07.233495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:07.251700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c194b70e-8ca5-43bf-933f-9743c69ad404 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:07.296585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:07.297297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:07.297587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:07.297868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:07.298725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:07.299041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:07.300560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:07.323848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 342/689] 10.222.0.20 () {56 vars in 1135 bytes} [Tue Jul 5 09:58:07 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:07.329255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:07.335911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:07.366153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8209edb-2c4b-406f-b9e5-312ddd88b5e2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:07.391768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:07.413119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:07.413812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:07.414099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:07.414379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:07.415115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:07.415429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:07.417085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:07.447752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 348/690] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:07.453466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:07.472447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2339e-3653-4de9-913c-0fa683fbe11e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:07.503830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:07.504588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:07.504863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:07.505147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:07.577398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:07.578932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:07.581759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:07.691065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7973d516-116d-44da-b9cd-85ef1b52aa85 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:07.722759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 343/691] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 05 09:58:07.727875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:07.758410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:07.759279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:07.759658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:07.760028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:07.770611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:07.773998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:07.794988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:07.796246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:07.797458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:07.808952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 349/692] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:58:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:07.813131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:07.894437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:07.904541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2627b2-40cd-48a5-ad49-2d7e12f8baee None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:07.906861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:07.969603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:07.970557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:07.971693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:07.972066 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:07.989976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:07.992476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:08.005802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd8fdd8-ceef-4555-8f1a-9aec2939d9c2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:08.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:08.025050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:08.025929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:08.025929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:08.025929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:08.025929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:08.030310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 344/693] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:08.035501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:08.070673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19841439-589c-4d25-8f7f-772853589140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:08.071578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19841439-589c-4d25-8f7f-772853589140 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:08.071859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19841439-589c-4d25-8f7f-772853589140 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:08.072130 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19841439-589c-4d25-8f7f-772853589140 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:08.072990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19841439-589c-4d25-8f7f-772853589140 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:08.073298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19841439-589c-4d25-8f7f-772853589140 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:08.075240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19841439-589c-4d25-8f7f-772853589140 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:08.110752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 350/694] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:58:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:08.116086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19841439-589c-4d25-8f7f-772853589140 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:08.212061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126a6029-ecbe-4e6f-b4e2-d6e9c9ba303c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:08.260733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:08.262200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:08.262485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:08.262759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:08.263484 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:08.263789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:08.265358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:08.287430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 345/695] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:08.292725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:08.295731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19841439-589c-4d25-8f7f-772853589140 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:08.300713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:08.322419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:08.388709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2bd42c-fe73-45ac-8a2f-8c30c0f32b5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:08.389438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2bd42c-fe73-45ac-8a2f-8c30c0f32b5c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:08.389727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2bd42c-fe73-45ac-8a2f-8c30c0f32b5c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:08.390008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2bd42c-fe73-45ac-8a2f-8c30c0f32b5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:08.426541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6895d94-e0d7-4433-90c2-4a0b83278c55 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:08.477490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:08.478316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:08.484425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:08.484425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:08.489514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:08.490996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:08.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:08.510677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:08.512103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:08.520741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 346/696] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:08.521573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:08.613376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:08.641160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26986b44-c432-4c3f-a499-2a3e5483cf8b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:08.669339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2bd42c-fe73-45ac-8a2f-8c30c0f32b5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:08.670902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2bd42c-fe73-45ac-8a2f-8c30c0f32b5c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:08.677397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:08.678298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:08.678815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:08.679258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:08.680215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:08.680702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:08.682272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:08.695726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 347/697] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:08.701137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:08.802980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267b95b7-fd43-4b99-b641-366b0b0fd381 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:08.838587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:08.839562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:08.840078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:08.840778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:08.843730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=c1d16fdc498e4d84bf8657c7f55d6b26) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:08.844276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:08.845929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:08.848591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:08.859065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 348/698] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:08 2022] GET /identity/v3/role_assignments?user.id=c1d16fdc498e4d84bf8657c7f55d6b26&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:08.978210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:08.989083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:08.993468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 351/699] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 877 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:58:08.995016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59b68fb-3d0e-47ac-87c0-2a5835a6ddcf None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:08.995310 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59b68fb-3d0e-47ac-87c0-2a5835a6ddcf None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:08.995591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59b68fb-3d0e-47ac-87c0-2a5835a6ddcf None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:08.997308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 352/700] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:09.003670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0411b6-d876-4b92-852c-2941f2ff719c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.003968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0411b6-d876-4b92-852c-2941f2ff719c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.004254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0411b6-d876-4b92-852c-2941f2ff719c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.055069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9c0411b6-d876-4b92-852c-2941f2ff719c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:09.058339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9c0411b6-d876-4b92-852c-2941f2ff719c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:09.160275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 353/701] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:09.160917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bc52b6-91d4-48be-9321-c0a63a4f0905 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.161209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bc52b6-91d4-48be-9321-c0a63a4f0905 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.161494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bc52b6-91d4-48be-9321-c0a63a4f0905 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.162878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 354/702] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:09.176905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fd2de3-a731-4720-a33e-b68d63980dde None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.177166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fd2de3-a731-4720-a33e-b68d63980dde None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.177706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fd2de3-a731-4720-a33e-b68d63980dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.189939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926fae5c-04f6-4da3-946c-0f3d033dbe00 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.211983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d2fd2de3-a731-4720-a33e-b68d63980dde None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:09.213632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d2fd2de3-a731-4720-a33e-b68d63980dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:09.242390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.244570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.244570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.244570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.257436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.258906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:09.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:09.280311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:09.281283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:09.281283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:09.281283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:09.281283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:09.281283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 349/703] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:09.282014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.310370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 355/704] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:58:09.311794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fd2de3-a731-4720-a33e-b68d63980dde None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.335343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:09.390971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177444f4-df7b-4a07-9818-b25d49d39965 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.400882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:09.428109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.429272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.429698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.430080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.430936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.432597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:09.448529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:09.449741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:09.450772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:09.452687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fd2de3-a731-4720-a33e-b68d63980dde None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.453177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 350/705] 10.222.0.20 () {56 vars in 1129 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:09.454135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.495393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.496896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.497354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.497782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.499023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:09.499477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.501675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.515706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 356/706] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:09.516804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.570326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2881446f-6133-48d2-a45d-645bd9dfe083 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.585272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:09.608047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.608885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.609165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.609434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.620992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.622907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740a83db-bf8c-4867-b985-b035e1b407ad None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.623653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:09.643051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:09.644036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:09.645332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:09.645332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:09.645332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:09.645332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:09.645332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 351/707] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:09.645332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.646569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:09.666347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ec0f2f-9a20-4472-9049-eccb83b1051c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.666911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec0f2f-9a20-4472-9049-eccb83b1051c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.667196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec0f2f-9a20-4472-9049-eccb83b1051c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.667476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ec0f2f-9a20-4472-9049-eccb83b1051c None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.668829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec0f2f-9a20-4472-9049-eccb83b1051c None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:09.669140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec0f2f-9a20-4472-9049-eccb83b1051c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.670517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ec0f2f-9a20-4472-9049-eccb83b1051c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.675779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 357/708] 10.222.0.20 () {56 vars in 1124 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:09.678832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193c26c-8594-4a92-8387-ed4bf3cb6b27 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.679339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193c26c-8594-4a92-8387-ed4bf3cb6b27 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.680068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8193c26c-8594-4a92-8387-ed4bf3cb6b27 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.681951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 358/709] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:09 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:09.682869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8193c26c-8594-4a92-8387-ed4bf3cb6b27 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.749991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee464140-3759-4ee0-94b3-3dea28897a44 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.785915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.786705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.786993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.787276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.790885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8193c26c-8594-4a92-8387-ed4bf3cb6b27 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.794737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.796180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:09.810614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:09.811571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:09.812498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 352/710] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:09.812977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.831490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e323ed76-6880-419d-a7ed-944951a7091e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.832469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323ed76-6880-419d-a7ed-944951a7091e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.832929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323ed76-6880-419d-a7ed-944951a7091e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.833396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e323ed76-6880-419d-a7ed-944951a7091e None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.834472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e323ed76-6880-419d-a7ed-944951a7091e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:09.834785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e323ed76-6880-419d-a7ed-944951a7091e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.836152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e323ed76-6880-419d-a7ed-944951a7091e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.843272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:09.854224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 359/711] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:09.854224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1598c083-d67c-43c2-8e82-ec96c310cfa4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.854224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1598c083-d67c-43c2-8e82-ec96c310cfa4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.854224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1598c083-d67c-43c2-8e82-ec96c310cfa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.869420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1598c083-d67c-43c2-8e82-ec96c310cfa4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:09.870604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1598c083-d67c-43c2-8e82-ec96c310cfa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:09.912614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:09.929494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a18ef0-f61e-4246-b6fc-143e3e1452c4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:09.956410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 360/712] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:09.956410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1598c083-d67c-43c2-8e82-ec96c310cfa4 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:09.968682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:09.969391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:09.969680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:09.969958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:09.970722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:09.971058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:09.972489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:09.978460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 353/713] 10.222.0.20 () {56 vars in 1124 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:09.984579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:10.061485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1598c083-d67c-43c2-8e82-ec96c310cfa4 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:10.098611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:10.099404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:10.099759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:10.100108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:10.101019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:10.101412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:10.103128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:10.122709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:10.123854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 361/714] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:10.125277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:10.130109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60d0489-24e8-4946-9789-5f9626efe986 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:10.167559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:10.172884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:10.173693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:10.173982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:10.174258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:10.174971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:10.176462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:10.209265 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 354/715] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:58:09 2022] POST /identity/v3/users => generated 254 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:10.210100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:10.286884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0272aeda-e7ed-40b3-8416-e4dfda1c2d93 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:10.343957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:10.344832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:10.345116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:10.345387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] PATH_INFO: `/v3/services/network` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:10.346054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:10.347436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:10.363817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:10.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:10.365861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 362/716] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:10 2022] GET /identity/v3/services/network => generated 88 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:10.372929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:10.397923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6274d3-8f60-4055-9c2e-38834ced471f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:10.455049 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:10.456192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:10.456522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:10.456808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:10.457932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=f99a9fe4b40646d389c3ebf47358956c) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:10.458253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:10.460532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:10.469714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 355/717] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:10 2022] GET /identity/v3/role_assignments?user.id=f99a9fe4b40646d389c3ebf47358956c&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:10.500762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8992baaa-7f49-489a-95c0-617a1315da82 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:10.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:10.541878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab80e9-acb1-4a0c-9358-315ab87ec613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:10.542611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab80e9-acb1-4a0c-9358-315ab87ec613 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:10.542946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab80e9-acb1-4a0c-9358-315ab87ec613 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:10.543230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab80e9-acb1-4a0c-9358-315ab87ec613 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:10.543974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab80e9-acb1-4a0c-9358-315ab87ec613 None admin] RBAC: Adding query filter params (name=network) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:10.544287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab80e9-acb1-4a0c-9358-315ab87ec613 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:10.545748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab80e9-acb1-4a0c-9358-315ab87ec613 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:10.551730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 363/718] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:10 2022] GET /identity/v3/services?name=network => generated 124 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:10.557051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:10.573210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:10.672639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49dac08-0a68-4eaf-b6f7-c6adcd708a7d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:10.714133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc3640d-ea36-47b3-b5be-0c466be10997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:10.715682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3640d-ea36-47b3-b5be-0c466be10997 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:10.716169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3640d-ea36-47b3-b5be-0c466be10997 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:10.716693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3640d-ea36-47b3-b5be-0c466be10997 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:10.717635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3640d-ea36-47b3-b5be-0c466be10997 None admin] RBAC: Adding query filter params (type=network) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:10.718140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3640d-ea36-47b3-b5be-0c466be10997 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:10.719742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc3640d-ea36-47b3-b5be-0c466be10997 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:10.725689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 356/719] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:10 2022] GET /identity/v3/services?type=network => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:11.240680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8f3703-1303-4726-800b-6f453ee47969 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:11.241384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8f3703-1303-4726-800b-6f453ee47969 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:11.241877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8f3703-1303-4726-800b-6f453ee47969 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:11.243733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 364/720] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:11.253918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282d5ec-4ae3-4d9f-a266-672a9080ff1f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:11.254247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282d5ec-4ae3-4d9f-a266-672a9080ff1f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:11.254471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282d5ec-4ae3-4d9f-a266-672a9080ff1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:11.276551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-3282d5ec-4ae3-4d9f-a266-672a9080ff1f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:11.278282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3282d5ec-4ae3-4d9f-a266-672a9080ff1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:11.369383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 357/721] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:11.404917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8f3703-1303-4726-800b-6f453ee47969 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:11.541362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8f3703-1303-4726-800b-6f453ee47969 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:11.577126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:11.577910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:11.578197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:11.578472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:11.588826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:11.597068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:11.597974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:11.599033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 365/722] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:11.605489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3282d5ec-4ae3-4d9f-a266-672a9080ff1f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:11.614327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:11.769675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3282d5ec-4ae3-4d9f-a266-672a9080ff1f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:11.814573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:11.815171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:11.815455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:11.815731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:11.816505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:11.816816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:11.818192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:11.832327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 358/723] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:11.837471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:11.881993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:11.969007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32da86f-e54b-4ecc-a8ed-152606d3678e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:12.005815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:12.006616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.006901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.007178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.018997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:12.020481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:12.038275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:12.039250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:12.040519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:12.040519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:12.040519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:12.040519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:12.040519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 366/724] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:12.045566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:12.128950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:12.179462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fedd0c7-7377-4ec8-af7e-d3a492c5007f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:12.214291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:12.215038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.215365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.215646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.216462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:12.216777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:12.218160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:12.231020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 359/725] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:12.237470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:12.315698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:12.342132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbca2ebd-8680-410f-9723-33c3ae3708bd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:12.377974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa21413f-a773-493b-92e5-a0b3dad93efb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:12.378838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa21413f-a773-493b-92e5-a0b3dad93efb None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.379185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa21413f-a773-493b-92e5-a0b3dad93efb None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.379456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa21413f-a773-493b-92e5-a0b3dad93efb None admin] PATH_INFO: `/v3/users/swift` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.387659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa21413f-a773-493b-92e5-a0b3dad93efb None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:12.389148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa21413f-a773-493b-92e5-a0b3dad93efb None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-aa21413f-a773-493b-92e5-a0b3dad93efb None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:12.405110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:12.406499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:12.407951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 367/726] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:12 2022] GET /identity/v3/users/swift => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:12.413854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:12.526140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399408f7-4e1b-4138-a055-18b763b835dc None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.526463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399408f7-4e1b-4138-a055-18b763b835dc None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.526711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399408f7-4e1b-4138-a055-18b763b835dc None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.528718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 368/727] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:12.535870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d646f04a-d737-417f-8a2a-df3e5d40299d None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.536162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d646f04a-d737-417f-8a2a-df3e5d40299d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.536479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d646f04a-d737-417f-8a2a-df3e5d40299d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.551504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25356a-2453-4091-9058-73a920ad3eb8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:12.569466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d646f04a-d737-417f-8a2a-df3e5d40299d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:12.570683 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d646f04a-d737-417f-8a2a-df3e5d40299d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:12.581022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:12.589186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:12.590536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.590944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.591325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.592274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:12.592715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:12.594247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:12.616570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 360/728] 10.222.0.20 () {56 vars in 1135 bytes} [Tue Jul 5 09:58:12 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:12.618060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:12.696586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 369/729] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:58:12.739537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d646f04a-d737-417f-8a2a-df3e5d40299d None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:12.745339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e94399d-638f-48bf-9bc9-de02d90b80ed None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:12.782859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:12.783609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.783892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.784168 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.839279 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:12.857733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:12.867444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:12.869204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:12.925800 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d646f04a-d737-417f-8a2a-df3e5d40299d None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:12.983330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135a62c1-60b2-434e-9204-92f80ea6f4b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:12.984381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135a62c1-60b2-434e-9204-92f80ea6f4b6 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:12.984910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135a62c1-60b2-434e-9204-92f80ea6f4b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:12.985494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135a62c1-60b2-434e-9204-92f80ea6f4b6 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:12.986548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135a62c1-60b2-434e-9204-92f80ea6f4b6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:12.988107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135a62c1-60b2-434e-9204-92f80ea6f4b6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:13.005543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 370/730] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:12 2022] POST /identity/v3/services => generated 251 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:58:13.013737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bec45-7ef2-4f9f-a00e-7a192109713f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.014461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bec45-7ef2-4f9f-a00e-7a192109713f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.014981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999bec45-7ef2-4f9f-a00e-7a192109713f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.017019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 371/731] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:13 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:13.023525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6179abfa-cd93-4ff4-93c1-67a0fd83a133 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.024155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6179abfa-cd93-4ff4-93c1-67a0fd83a133 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.024662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6179abfa-cd93-4ff4-93c1-67a0fd83a133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.036426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 361/732] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 05 09:58:13.037916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.049189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-6179abfa-cd93-4ff4-93c1-67a0fd83a133 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:13.050575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6179abfa-cd93-4ff4-93c1-67a0fd83a133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:13.137834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:13.145161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 372/733] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:13 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 05 09:58:13.167833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1657ba6c-863e-4fe4-8a1b-b6b9a2bba90d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.184923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6179abfa-cd93-4ff4-93c1-67a0fd83a133 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.214336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45de3bc-48ed-4250-8b84-7857a9dd4870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.217356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45de3bc-48ed-4250-8b84-7857a9dd4870 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.217802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45de3bc-48ed-4250-8b84-7857a9dd4870 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.218567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45de3bc-48ed-4250-8b84-7857a9dd4870 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.234905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45de3bc-48ed-4250-8b84-7857a9dd4870 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:13.236748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45de3bc-48ed-4250-8b84-7857a9dd4870 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-c45de3bc-48ed-4250-8b84-7857a9dd4870 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:13.257482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:13.258598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:13.259496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:13.259496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:13.259496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:13.259496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:13.259496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 362/734] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:13.260172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d51d64-88b0-48f1-945f-5c8cd0170ecb None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.260284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d51d64-88b0-48f1-945f-5c8cd0170ecb None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.261816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d51d64-88b0-48f1-945f-5c8cd0170ecb None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.262816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 363/735] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:13.267370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d51d64-88b0-48f1-945f-5c8cd0170ecb None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.268502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:13.334187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6179abfa-cd93-4ff4-93c1-67a0fd83a133 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.370550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868ebc99-a73e-483e-9b6f-8487e418efed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.372410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ebc99-a73e-483e-9b6f-8487e418efed None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.372922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ebc99-a73e-483e-9b6f-8487e418efed None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.373417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ebc99-a73e-483e-9b6f-8487e418efed None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.384659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868ebc99-a73e-483e-9b6f-8487e418efed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-868ebc99-a73e-483e-9b6f-8487e418efed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:13.393603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:13.394541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:13.395468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:13.398220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 373/736] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:13.399639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e4959-e8ce-448c-9c11-4ce7c850e368 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.404423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e4959-e8ce-448c-9c11-4ce7c850e368 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.404579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d51d64-88b0-48f1-945f-5c8cd0170ecb None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.405182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e4959-e8ce-448c-9c11-4ce7c850e368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.424024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a10e4959-e8ce-448c-9c11-4ce7c850e368 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:13.425575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a10e4959-e8ce-448c-9c11-4ce7c850e368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:13.445752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.446432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.446716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.446994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.447719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:13.448028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:13.450562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:13.453379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:13.468478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 364/737] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:13.469732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.519562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 374/738] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:13 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:13.520779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10e4959-e8ce-448c-9c11-4ce7c850e368 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.581842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:13.587397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d31034-a0eb-406a-a01d-4347c8a1707a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.624645 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.626090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.626610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.627089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.628039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:13.628591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:13.630236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:13.639493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 365/739] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:13.640434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.656378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10e4959-e8ce-448c-9c11-4ce7c850e368 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.690741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e2c3a9-2d57-4e6b-b15b-f3725a9d462d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.691561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2c3a9-2d57-4e6b-b15b-f3725a9d462d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.691849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2c3a9-2d57-4e6b-b15b-f3725a9d462d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.692125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e2c3a9-2d57-4e6b-b15b-f3725a9d462d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.699355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e2c3a9-2d57-4e6b-b15b-f3725a9d462d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:13.701337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e2c3a9-2d57-4e6b-b15b-f3725a9d462d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:13.706449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d9e2c3a9-2d57-4e6b-b15b-f3725a9d462d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:13.716149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:13.717275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:13.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 375/740] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:13.718668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b107e0-e7b8-47bd-9bbb-198ef2fcfb4d None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.719578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b107e0-e7b8-47bd-9bbb-198ef2fcfb4d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.719578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b107e0-e7b8-47bd-9bbb-198ef2fcfb4d None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.720536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 376/741] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:13 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:13.721051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b107e0-e7b8-47bd-9bbb-198ef2fcfb4d None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.758429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a13289-f16a-4a19-926c-da18037f33fd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.795595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.796487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.796765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.797043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.807942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:13.816152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:13.817238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:13.818102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 366/742] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:13.819323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:13.829021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:13.834514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b107e0-e7b8-47bd-9bbb-198ef2fcfb4d None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.870742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bb68d-f377-4ab3-8cda-1c055af0c79f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.871803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bb68d-f377-4ab3-8cda-1c055af0c79f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.872234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bb68d-f377-4ab3-8cda-1c055af0c79f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.872705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bb68d-f377-4ab3-8cda-1c055af0c79f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.884051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268bb68d-f377-4ab3-8cda-1c055af0c79f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:13.885824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268bb68d-f377-4ab3-8cda-1c055af0c79f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-268bb68d-f377-4ab3-8cda-1c055af0c79f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:13.910377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:13.911315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:13.912203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:13.912203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:13.912203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:13.912203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:13.912203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 377/743] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:13.913072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718c83f-1b6d-47b8-9326-4e2fe13b0810 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.913385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718c83f-1b6d-47b8-9326-4e2fe13b0810 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.913699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718c83f-1b6d-47b8-9326-4e2fe13b0810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.936607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0ae5a-2e99-45ab-a71f-09fdca746426 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:13.937521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b718c83f-1b6d-47b8-9326-4e2fe13b0810 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:13.938707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b718c83f-1b6d-47b8-9326-4e2fe13b0810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:13.957360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:13.977504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:13.978170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:13.978456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:13.978733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:13.979521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:13.979832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:13.981913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:13.999467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 367/744] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:13.999467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.008780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 378/745] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:13 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 05 09:58:14.016500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718c83f-1b6d-47b8-9326-4e2fe13b0810 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.104322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:14.116222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1578c-dbf9-4ae4-b015-d66a48989fd0 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.124140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718c83f-1b6d-47b8-9326-4e2fe13b0810 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.154077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.154804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.155092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.155363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.156096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:14.156411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.157785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.159122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.159786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.160078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.160363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.161112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:14.161423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.162796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.167692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 368/746] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:14.169262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.185140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 379/747] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:14.185140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.221678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:14.288515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1aab88-f3f3-4252-bb02-0b3bf05a5519 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.291376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c694144-4858-47c3-86c8-e514c3eda6ac None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.327571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.328454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.328932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.329299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.330030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.331198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.331723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.331896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.332136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.332438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.333274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=c1d16fdc498e4d84bf8657c7f55d6b26) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:14.333584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.335056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.341566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 369/748] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/role_assignments?user.id=c1d16fdc498e4d84bf8657c7f55d6b26&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:14.342592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.352225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 380/749] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:14 2022] POST /identity/v3/services => generated 240 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:14.353439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.361428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:14.566036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e010aa6-f8e6-4046-b63d-e2736858236c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.570320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:14.577424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae71df-6a0b-4fb2-ba0d-216aeb46f46e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.608625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.609700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.610197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.610726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.623339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.623911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.624107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.624382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.624693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.626226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.635066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.637895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:14.647956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:14.648872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:14.649754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:14.649754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:14.649754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:14.649754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:14.656486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 370/750] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:14.658651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:14.662874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:14.663792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:14.664776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:14.665484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 381/751] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:14.673366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.753296 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:14.820507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bbb888-79a1-4f21-8dbd-e6284d29ea76 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.844299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717582e6-4ab9-4a92-a75d-2ccad594293d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:14.883748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.884345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.884656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.884936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.885666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:14.885972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.887394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.905944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 371/752] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:14.907452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:14.912354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:14.913301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:14.913778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:14.914264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:14.915300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:14.915802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:14.917446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:14.940730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:14.947043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 382/753] 10.222.0.20 () {56 vars in 1149 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:14.952736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.080968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8eae2e-1611-41df-a363-6898b2de6b3c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.139300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55fceff-ffb6-44de-a3c8-db9931fb6da0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.142807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.143884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.144169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.144469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.160907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.162586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:15.163488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:15.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:15.186295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:15.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:15.190055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4fab51-20ad-42c8-8af4-d1d53cabc98d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.190293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 372/754] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:15.190632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4fab51-20ad-42c8-8af4-d1d53cabc98d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.190923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4fab51-20ad-42c8-8af4-d1d53cabc98d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.191199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4fab51-20ad-42c8-8af4-d1d53cabc98d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.195573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.279706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4fab51-20ad-42c8-8af4-d1d53cabc98d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.281556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4fab51-20ad-42c8-8af4-d1d53cabc98d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.295954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:15.323323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd1e159-1444-4093-be90-73d90a546e15 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.357130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f40df1-16f6-448e-bfb7-1802c6b95d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.357870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f40df1-16f6-448e-bfb7-1802c6b95d20 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.358174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f40df1-16f6-448e-bfb7-1802c6b95d20 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.358465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f40df1-16f6-448e-bfb7-1802c6b95d20 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.359266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f40df1-16f6-448e-bfb7-1802c6b95d20 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:15.359573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f40df1-16f6-448e-bfb7-1802c6b95d20 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.362054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f40df1-16f6-448e-bfb7-1802c6b95d20 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.384083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 373/755] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:15.385764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe592cc-9039-4e0f-859e-f38ec2501c36 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.386259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe592cc-9039-4e0f-859e-f38ec2501c36 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.386876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe592cc-9039-4e0f-859e-f38ec2501c36 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.392759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 374/756] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:15.393540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe592cc-9039-4e0f-859e-f38ec2501c36 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.408573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 383/757] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:14 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:58:15.409794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f3f5d1-7c0f-453e-8697-d09806e003c3 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.410088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f3f5d1-7c0f-453e-8697-d09806e003c3 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.410370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f3f5d1-7c0f-453e-8697-d09806e003c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.429135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c6f3f5d1-7c0f-453e-8697-d09806e003c3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:15.430317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c6f3f5d1-7c0f-453e-8697-d09806e003c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:15.442903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:15.516417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 384/758] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:15.516417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f3f5d1-7c0f-453e-8697-d09806e003c3 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.546306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe592cc-9039-4e0f-859e-f38ec2501c36 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.584384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.585204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.585487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.585763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.587769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:15.598004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.599473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:15.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:15.618322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:15.619248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:15.619248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:15.619248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:15.619248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:15.619248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 375/759] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:15.620026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.647869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f3f5d1-7c0f-453e-8697-d09806e003c3 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.684579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.685756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.686243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.686788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.697663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.699995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.707382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:15.720204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:15.721288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:15.722204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:15.722204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:15.722204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:15.722204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:15.724336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 385/760] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:15.725419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.738009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ef2f61-2ae1-4ea2-b233-1747d119545a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.773359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.774167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.774451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.774724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.775370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.777065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:15.789837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:15.790771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:15.791595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:15.796322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 376/761] 10.222.0.20 () {56 vars in 1129 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:15.797180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.837110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5f54a1-3824-4148-ad97-eb147aa71d15 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.849583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:15.876838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.877895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.878393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.878887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.879825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:15.880330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.881982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.895093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 386/762] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:15.896320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:15.910375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93922d2e-ac90-40f3-9dd6-3745d923a67e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:15.954254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:15.954732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:15.955015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:15.955292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:15.956011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:15.956318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:15.957715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:15.961480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:15.976351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 377/763] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:15.977247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.007718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62130cf3-cb70-401c-8ab4-8cb5ad283227 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.044203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.045275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.045830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.046309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.047267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:16.047777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.049386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.054682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 387/764] 10.222.0.20 () {56 vars in 1124 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:16.055743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.085904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196d9f0a-96a2-4c4c-8c49-3d70e8a135c2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.097623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:16.123442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.124236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.124563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.124837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.132005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.133487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:16.148252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:16.149181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:16.150081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:16.153048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 378/765] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:16.153942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.168712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cc5bc-d672-4b94-bf67-b92ad2165347 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.193288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:16.206956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.208160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.208691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.209170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.216699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.218434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:16.233895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:16.234811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:16.235697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:16.238211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 388/766] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:16.239221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.269351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d3b19a-e3ec-4158-8e4b-4584d9999713 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.311114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.311988 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.312269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.312574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.313288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:16.313592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.314971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.324891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 379/767] 10.222.0.20 () {56 vars in 1124 bytes} [Tue Jul 5 09:58:16 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:16.325757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.353577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502bcc28-aec3-4889-9a34-d125564f31b0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.374303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:16.400849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.402091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.402946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.403225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.404297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:16.404646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.406728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.424572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 389/768] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:16.425726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.463251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98927fb8-7de2-4825-95c6-1e6f848774b3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.474472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:16.506063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.506728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.507009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.507282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.508069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:16.508377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.509825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.523203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 380/769] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:16.523870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.543424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a9acc5-62c0-42ec-ab8d-c544ff826237 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.579419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888e27d1-7f2d-4386-ab35-21bd75445fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.580081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e27d1-7f2d-4386-ab35-21bd75445fb3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.580370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e27d1-7f2d-4386-ab35-21bd75445fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.580682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e27d1-7f2d-4386-ab35-21bd75445fb3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.581476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888e27d1-7f2d-4386-ab35-21bd75445fb3 None admin] RBAC: Adding query filter params (interface=public, service_id=f0fc8e522a8348ef8264f3b7420a0556, region_id=RegionOne) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:16.581791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888e27d1-7f2d-4386-ab35-21bd75445fb3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.583160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888e27d1-7f2d-4386-ab35-21bd75445fb3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.594774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 390/770] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 09:58:16 2022] GET /identity/v3/endpoints?service_id=f0fc8e522a8348ef8264f3b7420a0556&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:16.594774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943d75e-8fd2-4d24-8159-bff4f317d9d2 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.594774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943d75e-8fd2-4d24-8159-bff4f317d9d2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.594774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943d75e-8fd2-4d24-8159-bff4f317d9d2 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.594774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 391/771] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:16 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:16.594774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2943d75e-8fd2-4d24-8159-bff4f317d9d2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.634822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:16.644584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a8302f-6fee-4a5e-a57e-5986427fc765 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.686050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.687043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.687399 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.687749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/f99a9fe4b40646d389c3ebf47358956c/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.689877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=f99a9fe4b40646d389c3ebf47358956c, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.691391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.706126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 381/772] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:58:16 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/f99a9fe4b40646d389c3ebf47358956c/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:58:16.707149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:16.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2943d75e-8fd2-4d24-8159-bff4f317d9d2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.728730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:16.760363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0399bdd-eb3f-4a41-b71d-2717980c6eab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.761157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0399bdd-eb3f-4a41-b71d-2717980c6eab None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.761561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0399bdd-eb3f-4a41-b71d-2717980c6eab None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.762457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0399bdd-eb3f-4a41-b71d-2717980c6eab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.763417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0399bdd-eb3f-4a41-b71d-2717980c6eab None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=154e830e318245158269c03fd2d9621a) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:16.763861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0399bdd-eb3f-4a41-b71d-2717980c6eab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.765411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0399bdd-eb3f-4a41-b71d-2717980c6eab None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.772188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 392/773] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:16 2022] GET /identity/v3/role_assignments?user.id=154e830e318245158269c03fd2d9621a&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:16.773702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f67fb9-ffba-46ec-8c91-aa76ef5dbc72 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.774055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f67fb9-ffba-46ec-8c91-aa76ef5dbc72 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.774442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f67fb9-ffba-46ec-8c91-aa76ef5dbc72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.846829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffce1cc7-0484-46cb-b3d9-8335e948d4d4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:16.895962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da88b6ef-3220-47d8-8d9e-08acdf4ee0ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:16.896852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b6ef-3220-47d8-8d9e-08acdf4ee0ac None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.897217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b6ef-3220-47d8-8d9e-08acdf4ee0ac None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.897569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b6ef-3220-47d8-8d9e-08acdf4ee0ac None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.898360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da88b6ef-3220-47d8-8d9e-08acdf4ee0ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:16.899887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da88b6ef-3220-47d8-8d9e-08acdf4ee0ac None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:16.907287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 382/774] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:16 2022] GET /identity/v3/services => generated 772 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:16.908864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984e55a1-cb07-46ee-a326-baa191959af0 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.909233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984e55a1-cb07-46ee-a326-baa191959af0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.909588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984e55a1-cb07-46ee-a326-baa191959af0 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.912105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 383/775] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:16.917189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d49273-3e2f-472f-b1c6-776e7c815d3e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:16.917476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d49273-3e2f-472f-b1c6-776e7c815d3e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:16.917758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d49273-3e2f-472f-b1c6-776e7c815d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:16.928221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-04f67fb9-ffba-46ec-8c91-aa76ef5dbc72 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:16.930335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-04f67fb9-ffba-46ec-8c91-aa76ef5dbc72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:16.946792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-10d49273-3e2f-472f-b1c6-776e7c815d3e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:16.948445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-10d49273-3e2f-472f-b1c6-776e7c815d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:16.966866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:17.029931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 393/776] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:17.040527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 384/777] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:17.054839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f67fb9-ffba-46ec-8c91-aa76ef5dbc72 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.062315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d49273-3e2f-472f-b1c6-776e7c815d3e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.177991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:17.180455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d49273-3e2f-472f-b1c6-776e7c815d3e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:17.181891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f67fb9-ffba-46ec-8c91-aa76ef5dbc72 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:17.222015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:17.222943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:17.223327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:17.223702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] PATH_INFO: `/v3/services/network` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:17.224550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:17.226026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:17.226781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:17.227734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:17.228468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:17.228906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:17.241134 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:17.243009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:17.244155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:17.245179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:17.252492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 385/778] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/services/network => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:17.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:17.254027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:17.255031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:17.256292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.257539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 394/779] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:17.263964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.278940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:17.369611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f1f84e-eea9-43fb-8345-94fd90f89029 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:17.425822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a71d8d-17de-46f6-bd4e-86fdc08ed9c3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:17.434225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:17.434913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:17.435194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:17.435473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:17.436190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] RBAC: Adding query filter params (name=network) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:17.436524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:17.437922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:17.452131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 386/780] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/services?name=network => generated 124 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:17.457689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.485988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:17.486588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:17.486872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:17.487145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:17.487899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:17.488208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:17.489620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:17.504426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 395/781] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:17.510268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.552745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:17.582371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:17.616516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee165a-9dbd-4e61-ae47-a85d61e8eef8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:17.655412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:17.656220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:17.656601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:17.656919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:17.657770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] RBAC: Adding query filter params (type=network) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:17.658083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:17.660613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:17.665374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 387/782] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/services?type=network => generated 350 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:17.672109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.699928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0528d19-bd7e-46ec-89ca-2a4a4ec7b465 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:17.767006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:17.767723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:17.768257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:17.768580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:17.789908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:17.794366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:17.831995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:17.833008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:17.834577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:17.834577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:17.834577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:17.834577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:17.834577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 396/783] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:17.838939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:17.995391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:18.046109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed8e491-eeeb-4e2a-9371-b9bc74bd7804 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:18.104170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf604c7-13f9-4f1a-a46b-429b8aebc259 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:18.109582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:18.110273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:18.110560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:18.110838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:18.111590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:18.111900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:18.113355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:18.139158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 397/784] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:18.144459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:18.160326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:18.161020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:18.161304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:18.161581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:18.162379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] RBAC: Adding query filter params (interface=public, service_id=20fb430fb7cb47aea5588f776aab36f8, region_id=RegionOne) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:18.162694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:18.164072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:18.178547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 388/785] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 09:58:17 2022] GET /identity/v3/endpoints?service_id=20fb430fb7cb47aea5588f776aab36f8&interface=public®ion_id=RegionOne => generated 194 bytes in 507 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:18.185966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:18.190723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:18.352966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace58634-f52e-4c9d-b8f8-6d437c1a2503 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:18.353270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356340b-03a6-42a7-be9b-e4ea2f9f54e7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:18.390167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:18.390851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:18.391139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:18.391416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:18.392111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:18.393814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:18.398367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:18.399172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:18.399459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:18.399739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] PATH_INFO: `/v3/users/swift` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:18.399911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 389/786] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:18 2022] GET /identity/v3/services => generated 772 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:18.412807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:18.414296 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:18.430754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:18.431767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:18.433156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 398/787] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:18.434950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:18.439190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:18.535724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:18.576364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c007f9-e617-4505-84b4-c7f59613a796 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:18.613197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:18.614092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:18.614544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:18.614968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:18.615959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:18.616471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:18.618052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:18.644593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 390/788] 10.222.0.20 () {56 vars in 1135 bytes} [Tue Jul 5 09:58:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:18.649861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:18.757753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c47644-c0ea-4725-be71-8af36bd466df None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:18.805255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:18.806731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:18.807169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:18.807595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:18.828489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:18.830304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:18.852050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:18.853208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:18.854319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:18.854319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:18.854319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:18.854319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:18.856802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 399/789] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:18.862098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.026567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbe7c32-50b5-4d5a-81c3-d41854ce54d3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.064036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:19.075438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b192bee-b6c1-4324-9102-79a87e1d6316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.076109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b192bee-b6c1-4324-9102-79a87e1d6316 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.076401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b192bee-b6c1-4324-9102-79a87e1d6316 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.076682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b192bee-b6c1-4324-9102-79a87e1d6316 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.077405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b192bee-b6c1-4324-9102-79a87e1d6316 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:19.077707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b192bee-b6c1-4324-9102-79a87e1d6316 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:19.079072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b192bee-b6c1-4324-9102-79a87e1d6316 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:19.095641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 391/790] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:19.101360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.160197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1baa4aa4-901b-4fe4-8f0e-6cb24a7d78ec None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.161164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1baa4aa4-901b-4fe4-8f0e-6cb24a7d78ec None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.161652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1baa4aa4-901b-4fe4-8f0e-6cb24a7d78ec None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.163513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 392/791] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:19.169283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b181b-7e7e-4a1f-a2f3-c3ca21b4f402 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.171866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b181b-7e7e-4a1f-a2f3-c3ca21b4f402 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.172353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639b181b-7e7e-4a1f-a2f3-c3ca21b4f402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.197989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-639b181b-7e7e-4a1f-a2f3-c3ca21b4f402 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:19.199648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-639b181b-7e7e-4a1f-a2f3-c3ca21b4f402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:19.202174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:19.247567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237e728-0842-48e0-bace-414adb0f4577 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.271735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 393/792] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:19.277474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73af6d2-44ef-4195-8b16-874ddf430b66 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.277977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73af6d2-44ef-4195-8b16-874ddf430b66 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.278483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73af6d2-44ef-4195-8b16-874ddf430b66 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.280190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 394/793] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:19 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:19.286788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.287584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.287865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.288138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.288927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9505630f-aed4-40d0-ae05-9fe8828a91d8 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.289422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9505630f-aed4-40d0-ae05-9fe8828a91d8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.289895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9505630f-aed4-40d0-ae05-9fe8828a91d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.295648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:19.297223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:19.317373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9505630f-aed4-40d0-ae05-9fe8828a91d8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:19.318715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:19.319599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:19.320504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 400/794] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:19.321400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.321814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9505630f-aed4-40d0-ae05-9fe8828a91d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:19.392940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 395/795] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:19.394482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9505630f-aed4-40d0-ae05-9fe8828a91d8 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.446302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:19.456174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9538381c-8a23-40d7-a6fc-7957c94810dd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.472763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:19.493387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.494441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.494909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.495327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.504542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9505630f-aed4-40d0-ae05-9fe8828a91d8 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.506657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:19.514647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:19.515561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:19.516419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:19.523920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 401/796] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:19.525062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.542261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.543613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.543972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.544342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.545252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:19.545632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:19.547150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:19.564328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 396/797] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:19.564328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.642716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b73a2e-dbc9-4d1b-8a15-a5a71376b3c6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.671258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623f91c4-893a-499e-b3e8-db741e92a5e1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.678212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.679019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.679292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.679874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:19.690668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:19.699259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:19.700700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:19.701543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 402/798] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:19.702919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.717074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225c9e20-3ce6-4d7e-aa54-92ebdfb48f95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.717687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c9e20-3ce6-4d7e-aa54-92ebdfb48f95 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.717975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c9e20-3ce6-4d7e-aa54-92ebdfb48f95 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.718261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225c9e20-3ce6-4d7e-aa54-92ebdfb48f95 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.719022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225c9e20-3ce6-4d7e-aa54-92ebdfb48f95 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:19.719335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225c9e20-3ce6-4d7e-aa54-92ebdfb48f95 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:19.720737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225c9e20-3ce6-4d7e-aa54-92ebdfb48f95 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:19.729406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 397/799] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:19.731080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d24037a-8f6a-49c5-86f2-445b2d867588 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.731376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d24037a-8f6a-49c5-86f2-445b2d867588 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.731662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d24037a-8f6a-49c5-86f2-445b2d867588 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.736555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 398/800] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:19.736555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d24037a-8f6a-49c5-86f2-445b2d867588 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.738200 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:19.817464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea9d399-090a-4038-af7f-e5c9b0c5f2f8 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.846687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d24037a-8f6a-49c5-86f2-445b2d867588 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:19.853362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.854386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.854666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.854941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/c1d16fdc498e4d84bf8657c7f55d6b26/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.857054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=c1d16fdc498e4d84bf8657c7f55d6b26, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:19.858606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:19.880434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 403/801] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:58:19 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/c1d16fdc498e4d84bf8657c7f55d6b26/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:58:19.880434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:19.891669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a7e5d-ffc0-4198-b2cd-17b779d33fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:19.892352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a7e5d-ffc0-4198-b2cd-17b779d33fba None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.892665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a7e5d-ffc0-4198-b2cd-17b779d33fba None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.892940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a7e5d-ffc0-4198-b2cd-17b779d33fba None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.894094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6a7e5d-ffc0-4198-b2cd-17b779d33fba None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:19.894404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6a7e5d-ffc0-4198-b2cd-17b779d33fba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:19.896012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6a7e5d-ffc0-4198-b2cd-17b779d33fba None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:19.910818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 399/802] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:19.912527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f490674-12b1-47fa-93a2-d7c23f775e96 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:19.912816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f490674-12b1-47fa-93a2-d7c23f775e96 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:19.913093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f490674-12b1-47fa-93a2-d7c23f775e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:19.941806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2f490674-12b1-47fa-93a2-d7c23f775e96 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:19.943029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2f490674-12b1-47fa-93a2-d7c23f775e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:19.970875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:19.985124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:20.006670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee08041-0b07-4ca9-b823-9a453d4d807e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.022774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 400/803] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:20.024195 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f490674-12b1-47fa-93a2-d7c23f775e96 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.062129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.063345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.063827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.064352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.076814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.078573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:20.098802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:20.099751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:20.101295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:20.101295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:20.101295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:20.101295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:20.103890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 404/804] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:20.105112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.156120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f490674-12b1-47fa-93a2-d7c23f775e96 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.193384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.194195 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.194490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.194768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.206026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.207475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:20.226828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:20.227714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:20.229607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:20.229607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:20.229607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:20.229607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:20.229607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 401/805] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:20.233995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.241670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d4d15-56c7-4168-bb57-0e7f3e1cdf2a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.250101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:20.290212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.292007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.292579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.293280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.294114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.296574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:20.313300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:20.314502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 05 09:58:20.315552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:20.322328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 405/806] 10.222.0.20 () {56 vars in 1129 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:20.323294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.327147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:20.356729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1106c529-a577-40ed-88e7-64d6c1c251d9 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.395825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.396907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.397302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.397979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.399099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:20.399508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.401016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.416535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 402/807] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:20.421758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.450700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977ec1ea-3ffc-4a5c-a68c-11a95a1c1e03 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.500044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:20.502371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.503279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.503706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.504158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.506221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:20.506718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.509178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.526726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 406/808] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:20.527895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.537415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b855a-f04a-4f0e-a7b3-79a2ec61c29e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.585365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.586341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.587289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.587715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.588863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:20.589335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.591775 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.596450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:20.603613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 403/809] 10.222.0.20 () {56 vars in 1124 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:20.603613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.655493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7266d97-0a8d-438d-9a56-dfa9a87a6ba3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.697214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.698222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.698622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.699053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.706928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.708549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:20.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:20.725076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:20.726214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:20.729015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2e156c-f8bf-4bd4-ac48-1e140d37273a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.732714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 407/810] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:20.733721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.773487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.774515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.775069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.775349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.779500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:20.785824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.787871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:20.803313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:20.804259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:20.805237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 404/811] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:20.805833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.839554 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba99667-52cf-45c3-bfe6-cfd7ce76865e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.861433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:20.874123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a97fa06-897f-4b1f-b233-9920be49c2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.875034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97fa06-897f-4b1f-b233-9920be49c2c4 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.875486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97fa06-897f-4b1f-b233-9920be49c2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.875771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97fa06-897f-4b1f-b233-9920be49c2c4 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.876509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a97fa06-897f-4b1f-b233-9920be49c2c4 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:20.876822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a97fa06-897f-4b1f-b233-9920be49c2c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.878189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a97fa06-897f-4b1f-b233-9920be49c2c4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.888890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 408/812] 10.222.0.20 () {56 vars in 1124 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:20.888890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deeaa23a-b067-45b7-8017-ee4329d8982c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.888890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deeaa23a-b067-45b7-8017-ee4329d8982c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.888890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deeaa23a-b067-45b7-8017-ee4329d8982c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.891906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 409/813] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:20.892712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deeaa23a-b067-45b7-8017-ee4329d8982c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:20.917312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cb9be9-064f-42a8-a52e-a9cd6874392d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:20.955495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:20.956170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:20.956495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:20.956777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:20.957619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:20.957934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:20.959373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:20.981184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 405/814] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:20.981184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.011017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deeaa23a-b067-45b7-8017-ee4329d8982c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.042454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.050059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf3bd81-8f1e-48bb-a588-27d15a6ffd41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.050739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf3bd81-8f1e-48bb-a588-27d15a6ffd41 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.051024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf3bd81-8f1e-48bb-a588-27d15a6ffd41 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.051298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf3bd81-8f1e-48bb-a588-27d15a6ffd41 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.052135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf3bd81-8f1e-48bb-a588-27d15a6ffd41 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:21.052475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf3bd81-8f1e-48bb-a588-27d15a6ffd41 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.053906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf3bd81-8f1e-48bb-a588-27d15a6ffd41 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.076237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 410/815] 10.222.0.20 () {56 vars in 1149 bytes} [Tue Jul 5 09:58:20 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:21.076237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614e036a-0458-4ef3-84b8-e5a9de20073d None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.076237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614e036a-0458-4ef3-84b8-e5a9de20073d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.076237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614e036a-0458-4ef3-84b8-e5a9de20073d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.091517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-614e036a-0458-4ef3-84b8-e5a9de20073d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:21.092815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-614e036a-0458-4ef3-84b8-e5a9de20073d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:21.095000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e85db7-dd41-4cd6-b24d-a08bfef20fde None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.130452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.131057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.131346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.131625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.132363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.133764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.164791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 406/816] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:20 2022] POST /identity/v3/endpoints => generated 346 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:21.165905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.171196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 411/817] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:21.173967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614e036a-0458-4ef3-84b8-e5a9de20073d None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.303380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614e036a-0458-4ef3-84b8-e5a9de20073d None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.304216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43a6396-af8f-4d58-9a8b-e362da001f64 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.317253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.348664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.349334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.349624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.349906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.350923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.351718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.352003 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.352278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.365730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.367472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:21.386456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:21.387375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:21.388300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:21.388300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:21.388300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:21.388300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:21.388300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 412/818] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:21.388616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.425278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.441694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.443889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.506049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692162a-d0f6-4000-b5f7-a80a215a48a2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.541671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.542631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.543546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.544149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] PATH_INFO: `/v3/services/network` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.544649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.546106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:21.558440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:21.560092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:21.560955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 413/819] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/services/network => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:21.561760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.576761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 407/820] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:58:21.577838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.592941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.674036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.690563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3344585-9489-475f-af27-1de7a3b1b1f9 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.700311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f1993e-1f69-48cf-88a8-973c7178518a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.727897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.728507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.728899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.729281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.730109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:21.731236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.732782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.745682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.747098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.747512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.747922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.748798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] RBAC: Adding query filter params (name=network) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:21.749252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.750625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 414/821] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:21.750921 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.751686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.757566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 408/822] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/services?name=network => generated 124 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:21.758545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.854601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.862863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6b1a4d-cec4-470b-ad6e-13a75d109607 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.889612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38218f1e-1dc1-429a-8d1e-9df56bbdb101 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:21.900989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.901905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.902829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.903211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.913923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.915636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.931672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:21.938819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:21.939624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:21.939910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:21.940184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:21.945395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:21.946267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:21.947161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:21.947161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:21.947161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:21.947161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:21.947764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:21.948894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:21.949849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 415/823] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:21.950918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:21.966689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:21.967598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:21.968560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:21.972478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 409/824] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:21.973596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.068920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0468ba72-b70d-45e1-8b4c-00f4ff30e40d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.086972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32251c4-8405-4680-b222-ff1e6fdd90ac None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.106827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.107570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.107855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.108132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.108873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] RBAC: Adding query filter params (type=network) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:22.109180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.110813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.120902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 416/825] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/services?type=network => generated 350 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:22.121058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a0126a-4d56-4624-8e96-b0c05cf805a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.121543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0126a-4d56-4624-8e96-b0c05cf805a4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.121748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.121845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0126a-4d56-4624-8e96-b0c05cf805a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.122090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a0126a-4d56-4624-8e96-b0c05cf805a4 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.122807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a0126a-4d56-4624-8e96-b0c05cf805a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:22.123113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a0126a-4d56-4624-8e96-b0c05cf805a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.124537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a0126a-4d56-4624-8e96-b0c05cf805a4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.126946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:22.139882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 410/826] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:22.141237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf9e1c-c626-4f45-943d-53815395a397 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.141527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf9e1c-c626-4f45-943d-53815395a397 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.141801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abf9e1c-c626-4f45-943d-53815395a397 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.147380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 411/827] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:22.148156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abf9e1c-c626-4f45-943d-53815395a397 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.194840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:22.235659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19edf6db-aeae-4800-b0f6-d2a037382303 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.263577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abf9e1c-c626-4f45-943d-53815395a397 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.270173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.271155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.271155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.271427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.272152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:22.272492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.273911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.290522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 417/828] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:22.290522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.307193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ea8dce-70f3-43b7-a171-1d7b650cd57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.307881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ea8dce-70f3-43b7-a171-1d7b650cd57c None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.308143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ea8dce-70f3-43b7-a171-1d7b650cd57c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.308456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ea8dce-70f3-43b7-a171-1d7b650cd57c None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.309778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ea8dce-70f3-43b7-a171-1d7b650cd57c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.311147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ea8dce-70f3-43b7-a171-1d7b650cd57c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.330791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 412/829] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:22 2022] POST /identity/v3/endpoints => generated 327 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:22.332277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e8fbc-8670-4179-8275-b15171733e02 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.332739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e8fbc-8670-4179-8275-b15171733e02 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.333170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e8fbc-8670-4179-8275-b15171733e02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.353203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-484e8fbc-8670-4179-8275-b15171733e02 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:22.354996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-484e8fbc-8670-4179-8275-b15171733e02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:22.415858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160cc83d-aa2c-4ac9-a109-5addf4a268d5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.426343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:22.445010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:22.476418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 413/830] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:22 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:22.476418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484e8fbc-8670-4179-8275-b15171733e02 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.476733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.476733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.476733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.476733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.490072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.491576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:22.508360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:22.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:22.510359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 418/831] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:22.513065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.628688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484e8fbc-8670-4179-8275-b15171733e02 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.638978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c35362-17b6-4ffa-842c-4948e62f0a1c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.665291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.666237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.666466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.666738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/154e830e318245158269c03fd2d9621a/roles/0d7a50b0cac3467b8b2000f706f52329` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.679943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.680786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.681195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.681570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.682232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=154e830e318245158269c03fd2d9621a, role_id=0d7a50b0cac3467b8b2000f706f52329)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.683756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.692734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:22.706200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 414/832] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:58:22 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/154e830e318245158269c03fd2d9621a/roles/0d7a50b0cac3467b8b2000f706f52329 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:58:22.706200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:22.706461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:22.707355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 419/833] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:22.708196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.745105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:22.763847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:22.837572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c94b962-392c-4ebe-9a66-d3e36c1e3a4e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.856269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a824c6-c49e-4112-ad14-945b7359fc97 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:22.888621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.889606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.890342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.890624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.891581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:22.892196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.894191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.904421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:22.905658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:22.905953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:22.906242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:22.907507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:22.907507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:22.908933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:22.910385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 415/834] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:22.916329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:22.922304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 420/835] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:22.928038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.014441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:23.031982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a91ee3-ce67-4a36-8a6e-9c22051f1209 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.052750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:23.056852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eaff8d2-2939-46a3-86fc-43b9cd36fe44 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.070011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.070668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.070950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.071224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.072061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=f99a9fe4b40646d389c3ebf47358956c) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:23.072367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.073825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:23.080303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 416/836] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:22 2022] GET /identity/v3/role_assignments?user.id=f99a9fe4b40646d389c3ebf47358956c&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:23.098210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d2a2f-3621-415e-969a-b1e1fcbf72f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.099036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d2a2f-3621-415e-969a-b1e1fcbf72f2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.099323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d2a2f-3621-415e-969a-b1e1fcbf72f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.099601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d2a2f-3621-415e-969a-b1e1fcbf72f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.112197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0d2a2f-3621-415e-969a-b1e1fcbf72f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.113739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0d2a2f-3621-415e-969a-b1e1fcbf72f2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-fb0d2a2f-3621-415e-969a-b1e1fcbf72f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:23.133569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:23.134542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:23.135434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:23.135434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:23.135434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:23.135434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:23.135434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 421/837] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:23.140999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.253923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09785b52-1b4d-460b-bbc5-f157c7a94539 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.291444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.292114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.292418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.292715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.294058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:23.294368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.295812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:23.306866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7831b5dd-1f7e-4cb1-ad60-1a45964cdb42 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.307141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7831b5dd-1f7e-4cb1-ad60-1a45964cdb42 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.307394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7831b5dd-1f7e-4cb1-ad60-1a45964cdb42 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.308251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 417/838] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:23.309096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 422/839] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:23.313583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.314275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d352b88d-b2cc-4f49-9d2e-49360ce304a7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.314553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d352b88d-b2cc-4f49-9d2e-49360ce304a7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.314833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d352b88d-b2cc-4f49-9d2e-49360ce304a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.327924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:23.335050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d352b88d-b2cc-4f49-9d2e-49360ce304a7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:23.340765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d352b88d-b2cc-4f49-9d2e-49360ce304a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:23.427809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 423/840] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:23 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:23.438378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fe0267-a076-442a-a26f-b48292cf9713 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.449161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d352b88d-b2cc-4f49-9d2e-49360ce304a7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.475891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.476725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.477012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.477291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] PATH_INFO: `/v3/users/swift` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.486932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.488406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:23.503482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:23.504534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 05 09:58:23.505559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:23.508512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 418/841] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:23 2022] GET /identity/v3/users/swift => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:23.510658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.564099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:23.585647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d352b88d-b2cc-4f49-9d2e-49360ce304a7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.593008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:23.627664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5fbccd-ad71-401d-9444-8f8faa6ee1fa None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.631774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.632842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.633359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.633832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.634686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.636748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:23.652329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:23.653573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:23.654564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:23.657668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 424/842] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:23 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:23.663229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.671890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.672652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.672946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.673228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.674134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:23.674451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.675887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:23.699663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 419/843] 10.222.0.20 () {56 vars in 1135 bytes} [Tue Jul 5 09:58:23 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:23.700481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.779432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a4d2b3-ec52-4646-ae5c-823ebd848067 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.815572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.816247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.816604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.816900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.817617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:23.817932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.819314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:23.828758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 425/844] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:23 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:23.830835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:23.836797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86a64c-9976-4afe-ac64-0c6706614b67 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.845794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:23.875898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:23.889309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:23.890891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:23.891360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:23.891815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:23.960185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6233a8-ebfa-46ad-be6a-c12e37ca7261 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:23.981739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:23.983490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:24.005901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:24.007112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.007577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.008222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.009542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:24.010269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:24.013223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:24.019725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 426/845] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:23 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:24.102106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 420/846] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:23 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:58:24.109579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:24.194548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:24.223326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90ab4aa-3f41-43ac-93a5-2b4c977cef86 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:24.259531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:24.260336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.260665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.260942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.273412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:24.274910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:24.294143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:24.295096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:24.296006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:24.296006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:24.296006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:24.296006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:24.296006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 427/847] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:24.301024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:24.336780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:24.407303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f3b0c-7393-40b6-a0a1-f0bd580dcc7c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:24.443099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:24.443773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.444060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.444340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.445098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:24.445407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:24.446790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:24.459398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 421/848] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:24.466027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:24.572785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a520647-ee03-4175-b5f5-cba3099c6269 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:24.599368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:24.609432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:24.610241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.610531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.610811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.618016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:24.619490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:24.635225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:24.636135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:24.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 428/849] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:24.642655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:24.737615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:24.749878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e655fef-f5df-4cb9-b0e8-59c2fc1413b3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:24.785588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee180e3-9dbd-497b-8316-e31b216421ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:24.786254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee180e3-9dbd-497b-8316-e31b216421ea None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.786537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee180e3-9dbd-497b-8316-e31b216421ea None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.786874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee180e3-9dbd-497b-8316-e31b216421ea None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.787671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee180e3-9dbd-497b-8316-e31b216421ea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:24.788078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee180e3-9dbd-497b-8316-e31b216421ea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:24.789553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee180e3-9dbd-497b-8316-e31b216421ea None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:24.803152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 422/850] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:24.809575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:24.812228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb17d489-82d9-4fb9-8098-e1a8527aea88 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.812577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb17d489-82d9-4fb9-8098-e1a8527aea88 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.812858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb17d489-82d9-4fb9-8098-e1a8527aea88 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.818044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 423/851] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:24.819637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293a5ce-eef7-46d6-9a22-3a6b55d5798b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.819988 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293a5ce-eef7-46d6-9a22-3a6b55d5798b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.820333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8293a5ce-eef7-46d6-9a22-3a6b55d5798b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.837356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8293a5ce-eef7-46d6-9a22-3a6b55d5798b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:24.838534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8293a5ce-eef7-46d6-9a22-3a6b55d5798b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:24.912508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 424/852] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:24 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:24.932359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8293a5ce-eef7-46d6-9a22-3a6b55d5798b None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:24.945069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72587fd-7ee8-4f4e-9708-0fb927cfddf7 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:24.983098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:24.983739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:24.984019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:24.984295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:24.985154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=c1d16fdc498e4d84bf8657c7f55d6b26) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:24.985569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:24.987008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:24.993937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 429/853] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:24 2022] GET /identity/v3/role_assignments?user.id=c1d16fdc498e4d84bf8657c7f55d6b26&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:25.035907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:25.060254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8293a5ce-eef7-46d6-9a22-3a6b55d5798b None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.099243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f69432f-c9f5-4c53-b38d-601ca9c908af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.100063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69432f-c9f5-4c53-b38d-601ca9c908af None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.100344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69432f-c9f5-4c53-b38d-601ca9c908af None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.100650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69432f-c9f5-4c53-b38d-601ca9c908af None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.112648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f69432f-c9f5-4c53-b38d-601ca9c908af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-6f69432f-c9f5-4c53-b38d-601ca9c908af None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:25.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:25.121680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 09:58:25.122518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:25.127098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 425/854] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 09:58:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:25.128590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.185221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:25.227202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40220e5a-0a01-461a-af2e-b411dcb07341 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.227873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40220e5a-0a01-461a-af2e-b411dcb07341 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.228339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40220e5a-0a01-461a-af2e-b411dcb07341 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.230320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 426/855] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:25.236785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5f228-01bc-4b05-b516-30d770d08b45 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.237419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5f228-01bc-4b05-b516-30d770d08b45 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.237931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a5f228-01bc-4b05-b516-30d770d08b45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.242182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ceb18a7-367f-4a4d-a8e6-9f3e00a5a02a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.257615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-92a5f228-01bc-4b05-b516-30d770d08b45 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:25.258865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-92a5f228-01bc-4b05-b516-30d770d08b45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:25.286307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.287494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.287935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.288586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.289893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:25.290583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:25.292773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:25.307861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 430/856] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:25.310516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.341439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 427/857] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:25 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:25.367461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5f228-01bc-4b05-b516-30d770d08b45 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.418777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c62f84-2fb3-4fde-8142-adfc376d496f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.432460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:25.443753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:25.456444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.457447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.457908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.458358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.470062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:25.471695 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:25.480348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a5f228-01bc-4b05-b516-30d770d08b45 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:25.493894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:25.494964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:25.496064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:25.496064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:25.496064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:25.496064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:25.498760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 431/858] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:25.504432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.523225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.524452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.524944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.525557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.526607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:25.528821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:25.543108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:25.544582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:25.545424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:25.552548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 428/859] 10.222.0.20 () {56 vars in 1132 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:25.554634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.617146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f0b726-6df6-4a2e-8586-402b97143e19 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.663711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.664407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.664713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.665042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.665716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:25.666027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:25.667585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:25.674507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19602713-89fb-4cd8-a400-91348f696314 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.681586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 432/860] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:25.686749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.706212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:25.712224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.713271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.713755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.714259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.715697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:25.716201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:25.717852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:25.722489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:25.728059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 429/861] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:25.729291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.825426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e83a82e-0cc3-4268-9554-9e9c6abaed2d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.839022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135403f-723e-4a19-be0a-f38aa88afa20 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:25.872287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.873529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.873815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.874091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.878543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:25.879439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:25.879953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:25.880450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:25.881359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:25.881822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:25.882068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:25.883273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:25.883933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:25.895631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 430/862] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:25.899944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:25.900963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:25.901903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:25.906053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 433/863] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:25.907596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:25.975214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:26.023499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061a730b-6ab3-48fa-964c-3bc5615b2c6b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:26.060739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec7ba57-dcdf-4e3d-943b-85398e459731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:26.061413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec7ba57-dcdf-4e3d-943b-85398e459731 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:26.061700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec7ba57-dcdf-4e3d-943b-85398e459731 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:26.061969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec7ba57-dcdf-4e3d-943b-85398e459731 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:26.062808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec7ba57-dcdf-4e3d-943b-85398e459731 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:26.063113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec7ba57-dcdf-4e3d-943b-85398e459731 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:26.064558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec7ba57-dcdf-4e3d-943b-85398e459731 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:26.084642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 431/864] 10.222.0.20 () {56 vars in 1149 bytes} [Tue Jul 5 09:58:25 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:26.089155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:26.174963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbc5bd-63d3-4806-9264-28f4ea2694b5 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:26.175411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbc5bd-63d3-4806-9264-28f4ea2694b5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:26.175693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbc5bd-63d3-4806-9264-28f4ea2694b5 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:26.177888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 432/865] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:26.182348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d5edf-0279-475b-892d-1b748a0b7a70 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:26.182657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d5edf-0279-475b-892d-1b748a0b7a70 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:26.182938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699d5edf-0279-475b-892d-1b748a0b7a70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:26.196760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:26.200099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-699d5edf-0279-475b-892d-1b748a0b7a70 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:26.201334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-699d5edf-0279-475b-892d-1b748a0b7a70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:26.201942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7a3af7-4827-4839-b26a-c3e361b2b0d7 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:26.249528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:26.250223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:26.251052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:26.251342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:26.288706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 433/866] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:26 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:26.305319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699d5edf-0279-475b-892d-1b748a0b7a70 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:26.336596 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:26.338610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:26.361311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:26.425315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699d5edf-0279-475b-892d-1b748a0b7a70 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:26.455699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:26.461980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:26.462788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:26.463168 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:26.463673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:26.464615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:26.466618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:26.475654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 434/867] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:26 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:58:26.481694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:26.496603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 434/868] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:26 2022] POST /identity/v3/services => generated 251 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:26.595144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f8b53-5f4a-4688-b5bc-18246f3c7a75 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:26.639210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:26.640029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:26.640298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:26.640596 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:26.651935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:26.654077 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:26.673001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:26.673928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:26.674807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:26.674807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:26.674807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:26.674807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:26.674807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 435/869] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:26.680176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:26.788196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954d3889-82e4-4602-8bec-41c92170cef6 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:26.797740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:26.825506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:26.826173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:26.826458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:26.826734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:26.827452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:26.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:26.829157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:26.843006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 435/870] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:26.848494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:26.881453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:26.961330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bbf7dd-b4b7-4c58-81d3-777a422db109 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:27.000869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2dd43d-67b0-4a4b-a13b-64672400e973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:27.001675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2dd43d-67b0-4a4b-a13b-64672400e973 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.001961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2dd43d-67b0-4a4b-a13b-64672400e973 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.002241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2dd43d-67b0-4a4b-a13b-64672400e973 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.009360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2dd43d-67b0-4a4b-a13b-64672400e973 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:27.010819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2dd43d-67b0-4a4b-a13b-64672400e973 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-8a2dd43d-67b0-4a4b-a13b-64672400e973 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:27.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:27.026620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:27.027567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 436/871] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:27.032976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:27.139313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbcb177-6b8f-4f59-b448-746763beb08e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:27.174263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:27.175301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.175781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.176292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.177354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:27.177855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:27.179472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:27.182453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0ce09-4d12-4034-af39-15830ad19011 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.182453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0ce09-4d12-4034-af39-15830ad19011 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.182976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0ce09-4d12-4034-af39-15830ad19011 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.184871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 437/872] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:27.189712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c72f5-4956-4b97-b43d-184ce23aa319 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.190004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c72f5-4956-4b97-b43d-184ce23aa319 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.190294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67c72f5-4956-4b97-b43d-184ce23aa319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.194580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 436/873] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:27.203475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:27.207961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b67c72f5-4956-4b97-b43d-184ce23aa319 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:27.209192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b67c72f5-4956-4b97-b43d-184ce23aa319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:27.232354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:27.284519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 438/874] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:27 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:27.301965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67c72f5-4956-4b97-b43d-184ce23aa319 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:27.312870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0900a18f-d1bd-4802-9583-cbe6c5a599da None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:27.326880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:27.350261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:27.350927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.351205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.351484 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.352319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=154e830e318245158269c03fd2d9621a) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:27.352656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:27.354088 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:27.366961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 437/875] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:27 2022] GET /identity/v3/role_assignments?user.id=154e830e318245158269c03fd2d9621a&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:27.409645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67c72f5-4956-4b97-b43d-184ce23aa319 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:27.447928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:27.448787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.449073 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.449897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.450608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:27.452017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:27.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:27.466712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:27.468980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 439/876] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:27 2022] GET /identity/v3/services/object-store => generated 93 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:27.473336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:27.579750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:27.589767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d496d656-cb79-4fc6-9958-9d03c9bed0f2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:27.626499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ea73e6-0f1e-48b8-85e5-f616461e6f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:27.627544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ea73e6-0f1e-48b8-85e5-f616461e6f35 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.628016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ea73e6-0f1e-48b8-85e5-f616461e6f35 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.628504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ea73e6-0f1e-48b8-85e5-f616461e6f35 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.629454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ea73e6-0f1e-48b8-85e5-f616461e6f35 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:27.629946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ea73e6-0f1e-48b8-85e5-f616461e6f35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:27.631501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ea73e6-0f1e-48b8-85e5-f616461e6f35 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:27.636958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 438/877] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:27 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:27.642844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:27.749016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:27.752679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053d75b5-2ea7-4844-ab80-0f0b56dfbb4f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:27.791343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bd5274-10f4-4350-ad1c-3dabdddd6cfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:27.792023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bd5274-10f4-4350-ad1c-3dabdddd6cfa None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:27.792314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bd5274-10f4-4350-ad1c-3dabdddd6cfa None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:27.792628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bd5274-10f4-4350-ad1c-3dabdddd6cfa None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:27.793915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bd5274-10f4-4350-ad1c-3dabdddd6cfa None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:27.794226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bd5274-10f4-4350-ad1c-3dabdddd6cfa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:27.795608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bd5274-10f4-4350-ad1c-3dabdddd6cfa None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:27.800588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 440/878] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:27 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:28.018047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfa8cc-12bd-4bab-84ea-490256de46ed None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:28.018345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfa8cc-12bd-4bab-84ea-490256de46ed None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:28.018535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debfa8cc-12bd-4bab-84ea-490256de46ed None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:28.020469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 439/879] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:28.026225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55685172-fc5f-4dac-879a-2c76fad9857e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:28.026520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55685172-fc5f-4dac-879a-2c76fad9857e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:28.026799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55685172-fc5f-4dac-879a-2c76fad9857e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:28.050063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-55685172-fc5f-4dac-879a-2c76fad9857e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:28.051250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-55685172-fc5f-4dac-879a-2c76fad9857e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:28.120024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 441/880] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:28 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:28.147940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debfa8cc-12bd-4bab-84ea-490256de46ed None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:28.257763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debfa8cc-12bd-4bab-84ea-490256de46ed None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:28.295892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e208d8e1-3ac1-4633-b442-ef0ae5a5d64d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:28.296959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e208d8e1-3ac1-4633-b442-ef0ae5a5d64d None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:28.297472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e208d8e1-3ac1-4633-b442-ef0ae5a5d64d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:28.297977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e208d8e1-3ac1-4633-b442-ef0ae5a5d64d None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:28.298915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e208d8e1-3ac1-4633-b442-ef0ae5a5d64d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:28.300556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e208d8e1-3ac1-4633-b442-ef0ae5a5d64d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:28.317117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 440/881] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:28 2022] POST /identity/v3/services => generated 269 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:28.647718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86270a6-3a11-440a-9b70-5b436099738c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:28.648230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86270a6-3a11-440a-9b70-5b436099738c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:28.648230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86270a6-3a11-440a-9b70-5b436099738c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:28.649779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 442/882] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:28.655902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5c726-2934-4b17-8e41-ff1e79dc0db6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:28.656310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5c726-2934-4b17-8e41-ff1e79dc0db6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:28.656543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5c726-2934-4b17-8e41-ff1e79dc0db6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:28.674531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f2c5c726-2934-4b17-8e41-ff1e79dc0db6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:28.675711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f2c5c726-2934-4b17-8e41-ff1e79dc0db6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:28.686826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:28.751771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 441/883] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:28 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:28.774692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86270a6-3a11-440a-9b70-5b436099738c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:28.862848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:28.883965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86270a6-3a11-440a-9b70-5b436099738c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:28.919762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:28.920517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:28.920803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:28.921082 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:28.921734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:28.923174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:28.938509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:28.939625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:28.940492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 443/884] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:28 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:28.945915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c5c726-2934-4b17-8e41-ff1e79dc0db6 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:29.053587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c5c726-2934-4b17-8e41-ff1e79dc0db6 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:29.089696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:29.090939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.091226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.091504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.092222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:29.092561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:29.093955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:29.099361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 442/885] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:28 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:29.104664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:29.205638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435e492-8d45-4f88-a14d-b04b6e398f28 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:29.242844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:29.243501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.243787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.244062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.244786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:29.245091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:29.246985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:29.252344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 444/886] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:29 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:29.257790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:29.269951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:29.369187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb49709-9bd1-4f2f-84c6-0472dc7ddb99 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:29.405582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06838d4d-ccd4-411e-bb20-dcb983f13e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:29.406251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06838d4d-ccd4-411e-bb20-dcb983f13e6c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.406582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06838d4d-ccd4-411e-bb20-dcb983f13e6c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.406872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06838d4d-ccd4-411e-bb20-dcb983f13e6c None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.407675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06838d4d-ccd4-411e-bb20-dcb983f13e6c None admin] RBAC: Adding query filter params (interface=public, service_id=f235d655aa0147ccb2e82e4dc75f0125, region_id=RegionOne) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:29.407994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06838d4d-ccd4-411e-bb20-dcb983f13e6c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:29.409412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06838d4d-ccd4-411e-bb20-dcb983f13e6c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:29.415049 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 443/887] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 09:58:29 2022] GET /identity/v3/endpoints?service_id=f235d655aa0147ccb2e82e4dc75f0125&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:29.420965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:29.441203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:29.527167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3126550b-047f-4036-8110-b35c923dca72 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:29.563972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a727e8-ae3e-450d-88b3-e1fa9fa5d0f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:29.564668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a727e8-ae3e-450d-88b3-e1fa9fa5d0f5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.564953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a727e8-ae3e-450d-88b3-e1fa9fa5d0f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.565230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a727e8-ae3e-450d-88b3-e1fa9fa5d0f5 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.565918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a727e8-ae3e-450d-88b3-e1fa9fa5d0f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:29.567289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a727e8-ae3e-450d-88b3-e1fa9fa5d0f5 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:29.573547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 445/888] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:29 2022] GET /identity/v3/services => generated 1268 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 09:58:29.578865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9556b608-30a3-40be-a274-3e3341188caf None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.579155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9556b608-30a3-40be-a274-3e3341188caf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.579429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9556b608-30a3-40be-a274-3e3341188caf None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.588235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 444/889] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:29.591405 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468c90e7-dc3f-41ed-b657-9ef95893f3fb None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.591696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468c90e7-dc3f-41ed-b657-9ef95893f3fb None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.591972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468c90e7-dc3f-41ed-b657-9ef95893f3fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.610351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-468c90e7-dc3f-41ed-b657-9ef95893f3fb None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:29.611575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-468c90e7-dc3f-41ed-b657-9ef95893f3fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:29.688836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 446/890] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:29 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:29.712917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9556b608-30a3-40be-a274-3e3341188caf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:29.804652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:29.840633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9556b608-30a3-40be-a274-3e3341188caf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:29.876060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a47a7d-43e5-4b83-831a-804ce2fa5455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:29.876913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a47a7d-43e5-4b83-831a-804ce2fa5455 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.877194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a47a7d-43e5-4b83-831a-804ce2fa5455 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.877468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a47a7d-43e5-4b83-831a-804ce2fa5455 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.888890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a47a7d-43e5-4b83-831a-804ce2fa5455 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a4a47a7d-43e5-4b83-831a-804ce2fa5455 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:29.897341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:29.898267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:29.899158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 445/891] 10.222.0.20 () {56 vars in 1120 bytes} [Tue Jul 5 09:58:29 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:29.906833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468c90e7-dc3f-41ed-b657-9ef95893f3fb None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:29.910744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:29.917702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63d91f1-9c2e-4e0f-a635-7fbcafa96964 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.918015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63d91f1-9c2e-4e0f-a635-7fbcafa96964 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.926586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63d91f1-9c2e-4e0f-a635-7fbcafa96964 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.928457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 446/892] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:29 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:29.934006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e20e572-5a20-4d6e-85c1-e02a1873adeb None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:29.934297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e20e572-5a20-4d6e-85c1-e02a1873adeb None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:29.934582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e20e572-5a20-4d6e-85c1-e02a1873adeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:29.951556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7e20e572-5a20-4d6e-85c1-e02a1873adeb None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:29.952775 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7e20e572-5a20-4d6e-85c1-e02a1873adeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:30.028695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 447/893] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:29 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:30.032557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468c90e7-dc3f-41ed-b657-9ef95893f3fb None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.048962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e20e572-5a20-4d6e-85c1-e02a1873adeb None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.070284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:30.070927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.071212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.071529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.072288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:30.072627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:30.074067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:30.083595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 447/894] 10.222.0.20 () {56 vars in 1115 bytes} [Tue Jul 5 09:58:29 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:30.088836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.124834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:30.167972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:30.172718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e20e572-5a20-4d6e-85c1-e02a1873adeb None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.209412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:30.210451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.210939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.211410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.212330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:30.213850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:30.218039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2390a7be-ff68-4b90-be70-34cca4a82ac0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.229832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 448/895] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:30 2022] POST /identity/v3/services => generated 241 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:58:30.266615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:30.267859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.268320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.269162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.283068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:30.285391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:30.306764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:30.307721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:30.308752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:30.308752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:30.308752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:30.308752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:30.312188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 448/896] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:30.318302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.436010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f248bc-53e2-4c84-8185-3a0010be9bd6 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.462727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:30.479348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f507e903-fc54-4bd1-908f-afe3f20af334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:30.480017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f507e903-fc54-4bd1-908f-afe3f20af334 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.480326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f507e903-fc54-4bd1-908f-afe3f20af334 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.480651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f507e903-fc54-4bd1-908f-afe3f20af334 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.481378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f507e903-fc54-4bd1-908f-afe3f20af334 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:30.481693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f507e903-fc54-4bd1-908f-afe3f20af334 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:30.483129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f507e903-fc54-4bd1-908f-afe3f20af334 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:30.495724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 449/897] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:30.501290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.523270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e3a716-ad32-4d54-8e4b-c3b4a5097f0e None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.523873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e3a716-ad32-4d54-8e4b-c3b4a5097f0e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.524156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e3a716-ad32-4d54-8e4b-c3b4a5097f0e None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.525803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 450/898] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:30.533699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10c82fc-5b6a-41a0-ab47-cc7c504011cb None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.533979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10c82fc-5b6a-41a0-ab47-cc7c504011cb None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.534700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10c82fc-5b6a-41a0-ab47-cc7c504011cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.556802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d10c82fc-5b6a-41a0-ab47-cc7c504011cb None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:30.558258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d10c82fc-5b6a-41a0-ab47-cc7c504011cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:30.612893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1327f093-0c7b-4ef5-a080-08b382e6b8f1 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.626818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:30.649960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:30.650854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 451/899] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:30 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:30.653239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.653738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.654035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.662636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:30.664259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:30.675605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10c82fc-5b6a-41a0-ab47-cc7c504011cb None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:30.682216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:30.683124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:30.684067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:30.686918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 449/900] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 09:58:30 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:30.692927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.787045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10c82fc-5b6a-41a0-ab47-cc7c504011cb None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.801691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344ab34b-a265-4314-bc5d-c9d7aa5a5076 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.812891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:30.825663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:30.826869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.827329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.827778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.828736 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:30.830578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:30.837609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:30.838355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:30.838711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:30.839060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:30.840531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:30.840916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:30.842401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:30.846427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:30.847346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:30.848167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:30.856445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 452/901] 10.222.0.20 () {56 vars in 1132 bytes} [Tue Jul 5 09:58:30 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:30.857919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.866966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 450/902] 10.222.0.20 () {56 vars in 1149 bytes} [Tue Jul 5 09:58:30 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:30.868081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:30.898979 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:30.975309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95d1e59-0a53-4889-88f4-e7af79aeef1a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:30.993067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328e841c-6456-44b1-ab4d-6329fd053011 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.013530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.014474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.014928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.015340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.029666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.030250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.030534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.030809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.031520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:31.031830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.033492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.043805 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 453/903] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:30 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:31.046347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.095872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:31.101550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.103350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.144651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:31.175416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba457fa-8482-497d-b265-90b1cf458243 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.218373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.220040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.220574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.221194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.222421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:31.223175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.225219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.229290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 451/904] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:30 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:58:31.231697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 454/905] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:31.234929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.237679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.357890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0801af3a-2a5b-4b51-bec1-7d852fe76f13 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.370355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce678-767b-4480-948f-8ed4eb477eb8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.372807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:31.400335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:31.409645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.410708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.411088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.411464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.412980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.413940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.414420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.414996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.415993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] RBAC: Adding query filter params (interface=public, service_id=7b826af9393e4cd29e78e9887fd611e7, region_id=RegionOne) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:31.416556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.418713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.424176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.425588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 455/906] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/endpoints?service_id=7b826af9393e4cd29e78e9887fd611e7&interface=public®ion_id=RegionOne => generated 194 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:31.430291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.431533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:31.448989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:31.449988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:31.450872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:31.450872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:31.450872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:31.450872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:31.450872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 452/907] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:31.456756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.556106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d9a4c5-5178-463d-87ae-e5c04bbb65bd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.597765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.598443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.598724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.599087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.599677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.601089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.615653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 456/908] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/services => generated 1497 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 09:58:31.615854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b4f3b0-3b47-45ab-b1d1-7170727f122a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.654896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d61855e-97d8-4f3b-8b66-2b53368d20b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.655612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d61855e-97d8-4f3b-8b66-2b53368d20b7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.655909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d61855e-97d8-4f3b-8b66-2b53368d20b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.656234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d61855e-97d8-4f3b-8b66-2b53368d20b7 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.657074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d61855e-97d8-4f3b-8b66-2b53368d20b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:31.657385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d61855e-97d8-4f3b-8b66-2b53368d20b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.658799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d61855e-97d8-4f3b-8b66-2b53368d20b7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.672804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 453/909] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:31.678113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.692692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bfcfd3-9f55-416d-b64f-fc3bcdae226a None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.693041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bfcfd3-9f55-416d-b64f-fc3bcdae226a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.693567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bfcfd3-9f55-416d-b64f-fc3bcdae226a None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.694919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 454/910] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:31.700955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a6e090-abd2-42f7-9f81-4a7fdaeed1bb None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.701268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a6e090-abd2-42f7-9f81-4a7fdaeed1bb None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.701563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a6e090-abd2-42f7-9f81-4a7fdaeed1bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.719570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a5a6e090-abd2-42f7-9f81-4a7fdaeed1bb None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:31.719931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:31.721354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a5a6e090-abd2-42f7-9f81-4a7fdaeed1bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:31.775897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:31.810686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5912589b-86fc-48a7-90ce-017bd817787a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.818512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 455/911] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:31 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:31.843269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a6e090-abd2-42f7-9f81-4a7fdaeed1bb None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.849948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:31.850760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:31.851047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:31.851327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:31.858571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:31.860041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:31.874975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:31.875996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:31.877637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 457/912] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:31.882578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:31.970985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a6e090-abd2-42f7-9f81-4a7fdaeed1bb None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:31.999787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:32.014620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e35f943-af95-486f-a4b4-60818f1ad4e3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:32.015872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.016900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.017343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.017623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.018319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.019980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:32.035159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:32.036221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:32.037355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 456/913] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:32.045358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:32.065568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.066313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.066604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.066889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.067707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:32.068025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.069693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.085905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 458/914] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:32.092236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:32.118027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:32.171060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee63b8b1-12f4-4fe4-af6e-817ffd44b5a2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:32.207410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.208083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.208373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.208803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.209519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:32.209826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.211196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.221580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 457/915] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:32 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:32.222863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:32.231535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bedee6-e520-440a-8153-671f8c6455a5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:32.279544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.280886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.281387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.282114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.283825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] RBAC: Adding query filter params (role.id=5d677ad38b8c41579a9b215dadf5562e, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=154e830e318245158269c03fd2d9621a) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:32.284340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.286894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.304464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 459/916] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:32 2022] GET /identity/v3/role_assignments?user.id=154e830e318245158269c03fd2d9621a&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=5d677ad38b8c41579a9b215dadf5562e => generated 259 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:32.343068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:32.349309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385a1826-0145-4184-9eff-755660d87b1b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:32.389538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c246e0-afdb-44bf-a7bf-5c770244b890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.390285 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c246e0-afdb-44bf-a7bf-5c770244b890 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.390570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c246e0-afdb-44bf-a7bf-5c770244b890 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.390891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c246e0-afdb-44bf-a7bf-5c770244b890 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.391616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c246e0-afdb-44bf-a7bf-5c770244b890 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:32.391921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c246e0-afdb-44bf-a7bf-5c770244b890 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.393336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c246e0-afdb-44bf-a7bf-5c770244b890 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.403796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 458/917] 10.222.0.20 () {56 vars in 1114 bytes} [Tue Jul 5 09:58:32 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:32.405903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:32.424195 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:32.446412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fb234-275f-4c6c-9e29-b14ee7d1825e None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.446612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fb234-275f-4c6c-9e29-b14ee7d1825e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.446837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2fb234-275f-4c6c-9e29-b14ee7d1825e None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.448514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 459/918] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:32.456500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a49877-43f0-4f21-a082-9f54237909fe None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.456500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a49877-43f0-4f21-a082-9f54237909fe None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.456500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a49877-43f0-4f21-a082-9f54237909fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.474773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-72a49877-43f0-4f21-a082-9f54237909fe None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:32.476168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-72a49877-43f0-4f21-a082-9f54237909fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:32.536369 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75de7f73-d655-462e-9675-4c61a731f075 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:32.571539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 460/919] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:32 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:32.583376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.584308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.584829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.585308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.586261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.587941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.594020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a49877-43f0-4f21-a082-9f54237909fe None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:32.613253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 460/920] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:32 2022] POST /identity/v3/endpoints => generated 346 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:32.710826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:32.719371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a49877-43f0-4f21-a082-9f54237909fe None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:32.765725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.766538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.766822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.767099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.767755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.769436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:32.783359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:32.784379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:32.785251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:32.785987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 461/921] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:32 2022] GET /identity/v3/services/object-store => generated 93 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:32.790740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:32.870238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:32.904770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7d02c7-ebd2-47b3-824d-3525aa2e63f7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:32.940519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:32.941202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:32.941494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:32.941835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:32.942554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:32.942918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:32.944315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:32.949569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 461/922] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:32.955242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:33.061043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44204a75-355e-49ee-bfa3-7475b9d97938 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:33.099821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:33.100523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:33.100809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:33.101077 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:33.101790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:33.102094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:33.103470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:33.109595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 462/923] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:33.115210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:33.223814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0638ea3-4c3f-4016-b50a-51b42e35ad71 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:33.257994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355ff349-2cdd-486e-8707-1fcb52e42420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:33.258649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ff349-2cdd-486e-8707-1fcb52e42420 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:33.258927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ff349-2cdd-486e-8707-1fcb52e42420 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:33.259200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ff349-2cdd-486e-8707-1fcb52e42420 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:33.259982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355ff349-2cdd-486e-8707-1fcb52e42420 None admin] RBAC: Adding query filter params (interface=public, service_id=f0c73fe06e9f46c18267d7b9de0a9d77, region_id=RegionOne) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:33.260290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355ff349-2cdd-486e-8707-1fcb52e42420 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:33.261740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355ff349-2cdd-486e-8707-1fcb52e42420 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:33.263296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:33.267548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 462/924] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 09:58:33 2022] GET /identity/v3/endpoints?service_id=f0c73fe06e9f46c18267d7b9de0a9d77&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:33.273458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:33.299305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:33.384813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d408f167-8a04-4b68-b2b9-d16c191d2069 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:33.420142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6f15de-318f-45ed-bb95-ff140d059d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:33.420844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f15de-318f-45ed-bb95-ff140d059d73 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:33.421126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f15de-318f-45ed-bb95-ff140d059d73 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:33.421400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6f15de-318f-45ed-bb95-ff140d059d73 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:33.422083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6f15de-318f-45ed-bb95-ff140d059d73 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:33.423449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6f15de-318f-45ed-bb95-ff140d059d73 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:33.430049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 463/925] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:33 2022] GET /identity/v3/services => generated 1497 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 09:58:33.784213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e987ff2b-6878-4079-9b31-75f7a0a9fbc2 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:33.784619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e987ff2b-6878-4079-9b31-75f7a0a9fbc2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:33.784878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e987ff2b-6878-4079-9b31-75f7a0a9fbc2 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:33.786529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 463/926] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:33.791698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d3dc86-f1f9-4992-991d-215e53934ba7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:33.792180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d3dc86-f1f9-4992-991d-215e53934ba7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:33.792260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d3dc86-f1f9-4992-991d-215e53934ba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:33.809838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-07d3dc86-f1f9-4992-991d-215e53934ba7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:33.811109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-07d3dc86-f1f9-4992-991d-215e53934ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:33.882631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 464/927] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:33 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:33.906000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e987ff2b-6878-4079-9b31-75f7a0a9fbc2 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:34.026600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e987ff2b-6878-4079-9b31-75f7a0a9fbc2 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:34.061801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:34.062541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.062829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.063114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.063765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:34.065180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:34.080365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:34.081322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:34.082156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 464/928] 10.222.0.20 () {56 vars in 1132 bytes} [Tue Jul 5 09:58:33 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:34.089693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d3dc86-f1f9-4992-991d-215e53934ba7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:34.104977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:34.199075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d3dc86-f1f9-4992-991d-215e53934ba7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:34.233587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:34.234250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.234532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.234804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.235508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:34.235811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:34.237207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:34.242432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 465/929] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:34 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:34.248505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:34.313324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:34.355534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5d268a-28fc-4864-98be-933204b1b5d1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:34.391110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ae86c8-fa4c-41b2-934b-6c61c9cc965b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:34.391785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ae86c8-fa4c-41b2-934b-6c61c9cc965b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.392069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ae86c8-fa4c-41b2-934b-6c61c9cc965b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.392347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ae86c8-fa4c-41b2-934b-6c61c9cc965b None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.393084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ae86c8-fa4c-41b2-934b-6c61c9cc965b None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:34.393395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ae86c8-fa4c-41b2-934b-6c61c9cc965b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:34.395030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ae86c8-fa4c-41b2-934b-6c61c9cc965b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:34.400371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 465/930] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:34 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:34.405962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:34.486210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:34.513003 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca6894f-f1da-4c86-b4f7-e5cb34f03a3c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:34.526296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c4adf1-f7de-4e88-83a4-ab4cacfe62da None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.526592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c4adf1-f7de-4e88-83a4-ab4cacfe62da None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.526855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c4adf1-f7de-4e88-83a4-ab4cacfe62da None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.528426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 466/931] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:34.533207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edb364-35b3-4340-b4ce-996c4059b6c0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.533494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edb364-35b3-4340-b4ce-996c4059b6c0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.533776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edb364-35b3-4340-b4ce-996c4059b6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.551062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c1edb364-35b3-4340-b4ce-996c4059b6c0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:34.552286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c1edb364-35b3-4340-b4ce-996c4059b6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:34.553224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:34.554212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.554934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.556372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.557441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:34.559622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:34.595630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 466/932] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:34 2022] POST /identity/v3/endpoints => generated 347 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:34.639044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 467/933] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:34.663507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:34.773311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b00d199-45e3-4c98-a40e-71d2db1745e4 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:34.804808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:34.810431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8ccff3-5468-4f7f-9707-a91bf329627a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:34.811671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8ccff3-5468-4f7f-9707-a91bf329627a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.812518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8ccff3-5468-4f7f-9707-a91bf329627a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.812779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8ccff3-5468-4f7f-9707-a91bf329627a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.825614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8ccff3-5468-4f7f-9707-a91bf329627a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:34.833015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfc78c-056a-4be6-8b27-32564b2ed441 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-4d8ccff3-5468-4f7f-9707-a91bf329627a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:34.833273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:34.834167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:34.835178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:34.835882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfc78c-056a-4be6-8b27-32564b2ed441 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.835882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52bfc78c-056a-4be6-8b27-32564b2ed441 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.836015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 467/934] 10.222.0.20 () {56 vars in 1120 bytes} [Tue Jul 5 09:58:34 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:34.837003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 468/935] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:34.840936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bfc78c-056a-4be6-8b27-32564b2ed441 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:34.841417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af473576-b2ae-4e9a-99b2-0c644bf3b2f6 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:34.841712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af473576-b2ae-4e9a-99b2-0c644bf3b2f6 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:34.841989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af473576-b2ae-4e9a-99b2-0c644bf3b2f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:34.858901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-af473576-b2ae-4e9a-99b2-0c644bf3b2f6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:34.860004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-af473576-b2ae-4e9a-99b2-0c644bf3b2f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:34.900046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:34.963624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 468/936] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:34.971482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52bfc78c-056a-4be6-8b27-32564b2ed441 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:34.982834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af473576-b2ae-4e9a-99b2-0c644bf3b2f6 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.010560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.012759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.013231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.013713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.015097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:35.015727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.017743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.029621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 469/937] 10.222.0.20 () {56 vars in 1115 bytes} [Tue Jul 5 09:58:34 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:35.034339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.097840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af473576-b2ae-4e9a-99b2-0c644bf3b2f6 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.123599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:35.137248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.138281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.138807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.139323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.140217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.142036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.143626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING keystone.server.flask.common [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 05 09:58:35.170325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.sql.core [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 05 09:58:35.170325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 05 09:58:35.170325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [parameters: {'id': '44d7af53c62c443b96f08cb9477ecd7b', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 05 09:58:35.170325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80269) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 05 09:58:35.171768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07878c95-54cc-48c6-ac49-eb94e52570c6 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 44d7af53c62c443b96f08cb9477ecd7b.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 44d7af53c62c443b96f08cb9477ecd7b. Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result = self._query(query) Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application conn.query(q) Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 05 09:58:35.179999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result.read() Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 05 09:58:35.180931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application next(self.gen) Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application yield resource Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application next(self.gen) Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.session.rollback() Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.force_reraise() Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise self.value Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application session.commit() Jul 05 09:58:35.186939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._prepare_impl() Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.session.flush() Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._flush(objects) Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application compat.raise_( Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application flush_context.execute() Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 05 09:58:35.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rec.execute(self) Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 05 09:58:35.188873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result = self._query(query) Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application conn.query(q) Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result.read() Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 05 09:58:35.192140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [parameters: {'id': '44d7af53c62c443b96f08cb9477ecd7b', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.194361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:35.196477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 44d7af53c62c443b96f08cb9477ecd7b. Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 469/938] 10.222.0.20 () {60 vars in 1140 bytes} [Tue Jul 5 09:58:34 2022] POST /identity/v3/projects => generated 262 bytes in 199 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:35.198891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.200664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:35.226678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.228006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.228717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.229484 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.243682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.245878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:35.269811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:35.270756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:35.271650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:35.271650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:35.271650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:35.271650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:35.274192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 470/939] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:35.280579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.312549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65af325b-b65b-4d72-ac0f-69e7911d8935 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.350201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.351375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.351826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.352245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.359807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.361666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:35.377197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:35.378124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:35.379030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.385949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 470/940] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:35.388171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.389600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e068e0-76e5-45c7-9775-8241af80ecdd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.428056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.428860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.429209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.429552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.430422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:35.430793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.432231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.434257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:35.453472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 471/941] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:35.454491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.486913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:35.514499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f2566b-d1c6-4fee-bb59-6905325a1d99 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.552374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501e1777-48b8-4b85-8138-4fe47c77a93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.553534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501e1777-48b8-4b85-8138-4fe47c77a93d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.554048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501e1777-48b8-4b85-8138-4fe47c77a93d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.554524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501e1777-48b8-4b85-8138-4fe47c77a93d None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.555341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501e1777-48b8-4b85-8138-4fe47c77a93d None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:35.555717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501e1777-48b8-4b85-8138-4fe47c77a93d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.557429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501e1777-48b8-4b85-8138-4fe47c77a93d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.562357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dfec18-f7b6-472d-9ea5-bae959d2761e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.583463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 471/942] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:35.583463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77303f59-8500-4fe2-a236-dde4d4150275 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.583463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77303f59-8500-4fe2-a236-dde4d4150275 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.583463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77303f59-8500-4fe2-a236-dde4d4150275 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.583463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 472/943] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:35.589646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae353ab-da78-4889-afef-7fb179c8b365 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.589936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae353ab-da78-4889-afef-7fb179c8b365 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.590824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae353ab-da78-4889-afef-7fb179c8b365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.605905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.606892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.607280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.607660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.609635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0ae353ab-da78-4889-afef-7fb179c8b365 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:35.611454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0ae353ab-da78-4889-afef-7fb179c8b365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:35.617791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.619450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:35.637767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:35.638808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:35.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:35.644476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 472/944] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:35.645436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.688604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 473/945] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:35 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:35.712208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae353ab-da78-4889-afef-7fb179c8b365 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.745962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:35.769705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:35.791831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dcf0d5-3b01-4785-be2f-e063dada7232 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.840483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.841162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.841445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.841716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.842619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:35.842934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.844474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.850387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae353ab-da78-4889-afef-7fb179c8b365 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:35.869389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 473/946] 10.222.0.20 () {56 vars in 1149 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:35.875119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.890411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:35.891274 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:35.891559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:35.891826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:35.892514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:35.894316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:35.908092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:35.909399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:35.910437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 474/947] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/services/object-store => generated 93 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:35.917652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:35.995006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd043e-46f5-4939-b8af-95309239001d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.032916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.034034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.034528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.035007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.038404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:36.048130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.049136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60772d8-2ab9-443f-bfbe-b6b09f57caa3 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.050177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.053573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:36.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:36.074038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:36.075035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:36.075035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:36.075035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:36.075035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:36.077285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 474/948] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:36.082751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:36.094010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.094956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.095440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.095915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.096755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:36.097128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.098557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.108500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 475/949] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:35 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:36.110031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:36.210428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ec84ff-a954-4bd7-89f0-ea3aa4608eae None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.242509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e11ac-1461-4bc5-9559-081f466aa0d1 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.260364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.261072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.261356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.261632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.262912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:36.263218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.264628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.277191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 475/950] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:36.282411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:36.290327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.291001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.291348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.291563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.292272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:36.292636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.294575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.304492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 476/951] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:36 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:36.305316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:36.333525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:36.367029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:36.417542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf8cc16-5529-416f-9647-246195688d5d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.421692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0986cc5b-b9af-4b83-9f95-5d2478fb9d22 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.458071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.458312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.459398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.459511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.459897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.460225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.460421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.460953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.461851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.464422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.470202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.471966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:36.489456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:36.490417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:36.491298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:36.493034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 477/952] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:36 2022] POST /identity/v3/endpoints => generated 349 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:36.499491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 476/953] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:36.500924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:36.593989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:36.627268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f98357-f499-4e9f-9337-63f2617c62d1 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.665969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c32b51-c66d-497d-bc9b-2b759503b782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.666644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c32b51-c66d-497d-bc9b-2b759503b782 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.666931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c32b51-c66d-497d-bc9b-2b759503b782 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.667208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c32b51-c66d-497d-bc9b-2b759503b782 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.668011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c32b51-c66d-497d-bc9b-2b759503b782 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:36.668355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c32b51-c66d-497d-bc9b-2b759503b782 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.669841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c32b51-c66d-497d-bc9b-2b759503b782 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.685793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 478/954] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:36.689426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:36.753853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f85bd84-8420-4730-866e-a716cf27d16d None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.754523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f85bd84-8420-4730-866e-a716cf27d16d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.755241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f85bd84-8420-4730-866e-a716cf27d16d None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.758019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 479/955] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:36.763519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98707f4-694f-428f-8788-266f970c67aa None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.763809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98707f4-694f-428f-8788-266f970c67aa None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.764092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98707f4-694f-428f-8788-266f970c67aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.783570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f98707f4-694f-428f-8788-266f970c67aa None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:36.784967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f98707f4-694f-428f-8788-266f970c67aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:36.805785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0cc7add-3cf8-42ad-bd7c-03ee3f6e5d24 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:36.824003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:36.845086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:36.846241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:36.846529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:36.846815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/154e830e318245158269c03fd2d9621a/roles/5d677ad38b8c41579a9b215dadf5562e` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:36.849792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=154e830e318245158269c03fd2d9621a, role_id=5d677ad38b8c41579a9b215dadf5562e)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:36.851342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:36.870787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 477/956] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:58:36 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/154e830e318245158269c03fd2d9621a/roles/5d677ad38b8c41579a9b215dadf5562e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:58:36.880585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 480/957] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:36 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:36.896659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:37.023795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7494aa-190c-4f1a-8aa9-55374e0dd007 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:37.062628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:37.063489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:37.063774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:37.064050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] PATH_INFO: `/v3/services/compute` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:37.064780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:37.066167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:37.078239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:37.079473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:37.080543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:37.083214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 478/958] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:36 2022] GET /identity/v3/services/compute => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:37.085860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98707f4-694f-428f-8788-266f970c67aa None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:37.147024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:37.187303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98707f4-694f-428f-8788-266f970c67aa None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:37.220815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf8b5d6-86b9-44b4-b38e-218b00d9c09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:37.221623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf8b5d6-86b9-44b4-b38e-218b00d9c09e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:37.221978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf8b5d6-86b9-44b4-b38e-218b00d9c09e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:37.222321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf8b5d6-86b9-44b4-b38e-218b00d9c09e None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:37.223107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf8b5d6-86b9-44b4-b38e-218b00d9c09e None admin] RBAC: Adding query filter params (name=compute) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:37.223478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf8b5d6-86b9-44b4-b38e-218b00d9c09e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:37.224945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf8b5d6-86b9-44b4-b38e-218b00d9c09e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:37.235250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 481/959] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:37 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:37.236724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:37.280982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:37.343059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201ce191-0c7a-48a3-98b6-f45c007d543d None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:37.378931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15894fb5-ed7f-4d36-883d-a8ffa8fb19aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:37.379928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15894fb5-ed7f-4d36-883d-a8ffa8fb19aa None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:37.380422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15894fb5-ed7f-4d36-883d-a8ffa8fb19aa None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:37.380895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15894fb5-ed7f-4d36-883d-a8ffa8fb19aa None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:37.381796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15894fb5-ed7f-4d36-883d-a8ffa8fb19aa None admin] RBAC: Adding query filter params (type=compute) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:37.382286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15894fb5-ed7f-4d36-883d-a8ffa8fb19aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:37.383842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15894fb5-ed7f-4d36-883d-a8ffa8fb19aa None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:37.389166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 479/960] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:37 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:37.813366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1029902b-cc6c-4529-b980-38960578c829 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:37.813790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1029902b-cc6c-4529-b980-38960578c829 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:37.813906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1029902b-cc6c-4529-b980-38960578c829 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:37.815530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 482/961] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:37.820643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b55d3-1aab-4f4d-9de1-34f206158751 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:37.820936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b55d3-1aab-4f4d-9de1-34f206158751 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:37.821218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58b55d3-1aab-4f4d-9de1-34f206158751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:37.838528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a58b55d3-1aab-4f4d-9de1-34f206158751 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:37.839700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a58b55d3-1aab-4f4d-9de1-34f206158751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:37.907507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 480/962] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:37 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:37.936600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1029902b-cc6c-4529-b980-38960578c829 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:38.044513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1029902b-cc6c-4529-b980-38960578c829 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:38.072061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:38.081890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:38.082874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:38.083439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:38.083933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:38.084928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:38.086583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:38.102679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.sql.core [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 05 09:58:38.102679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 05 09:58:38.102679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [parameters: {'user_id': '7709484b08074e57bb5065faab69d4aa', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 05 09:58:38.102679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80269) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result = self._query(query) Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application conn.query(q) Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result.read() Jul 05 09:58:38.111475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 05 09:58:38.112515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application next(self.gen) Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application yield resource Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application next(self.gen) Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.session.rollback() Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.force_reraise() Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise self.value Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application session.commit() Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 05 09:58:38.113537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._prepare_impl() Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.session.flush() Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._flush(objects) Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application compat.raise_( Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application flush_context.execute() Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rec.execute(self) Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 05 09:58:38.114508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 05 09:58:38.115440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result = self._query(query) Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application conn.query(q) Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application result.read() Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 05 09:58:38.116323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [parameters: {'user_id': '7709484b08074e57bb5065faab69d4aa', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:38.119283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 05 09:58:38.121212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 05 09:58:38.123950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 05 09:58:38.123950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 05 09:58:38.123950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 05 09:58:38.123950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 05 09:58:38.123950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 05 09:58:38.123950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:38.126847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 483/963] 10.222.0.20 () {60 vars in 1131 bytes} [Tue Jul 5 09:58:37 2022] POST /identity/v3/users => generated 160 bytes in 191 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:38.133784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b55d3-1aab-4f4d-9de1-34f206158751 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:38.188812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:38.237595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58b55d3-1aab-4f4d-9de1-34f206158751 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:38.274051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad23387-ddd0-45e0-9a11-8ed55e79135b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:38.275221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad23387-ddd0-45e0-9a11-8ed55e79135b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:38.275719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad23387-ddd0-45e0-9a11-8ed55e79135b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:38.276211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad23387-ddd0-45e0-9a11-8ed55e79135b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:38.283739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad23387-ddd0-45e0-9a11-8ed55e79135b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:38.285431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad23387-ddd0-45e0-9a11-8ed55e79135b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-3ad23387-ddd0-45e0-9a11-8ed55e79135b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:38.300162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:38.301141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 05 09:58:38.302079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:38.304638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 481/964] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:38 2022] GET /identity/v3/users/cinder => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:38.313809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:38.416524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c426233-d8a8-4544-980f-2270b5310ca6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:38.451199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4d17be-2675-482c-b007-ff99be4d0af7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:38.451869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d17be-2675-482c-b007-ff99be4d0af7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:38.452153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d17be-2675-482c-b007-ff99be4d0af7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:38.452499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d17be-2675-482c-b007-ff99be4d0af7 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:38.453307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4d17be-2675-482c-b007-ff99be4d0af7 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:38.453614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4d17be-2675-482c-b007-ff99be4d0af7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:38.455321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4d17be-2675-482c-b007-ff99be4d0af7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:38.504735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 484/965] 10.222.0.20 () {56 vars in 1101 bytes} [Tue Jul 5 09:58:38 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:38.647652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb8483-dba3-4c91-bb45-32fa8bc81b5a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:38.648042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb8483-dba3-4c91-bb45-32fa8bc81b5a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:38.648247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cb8483-dba3-4c91-bb45-32fa8bc81b5a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:38.649890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 482/966] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:38.656309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96cf10-27e7-407b-8f3d-455dd2a364dc None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:38.656665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96cf10-27e7-407b-8f3d-455dd2a364dc None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:38.656952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b96cf10-27e7-407b-8f3d-455dd2a364dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:38.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5b96cf10-27e7-407b-8f3d-455dd2a364dc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:38.678055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5b96cf10-27e7-407b-8f3d-455dd2a364dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:38.727123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:38.754237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 485/967] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:38 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:38.777504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cb8483-dba3-4c91-bb45-32fa8bc81b5a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:38.874544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:38.883132 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cb8483-dba3-4c91-bb45-32fa8bc81b5a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:38.917688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:38.918498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:38.920466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:38.920466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:38.920638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:38.922454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:38.939253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:38.940290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:38.941160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 483/968] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:38.947901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b96cf10-27e7-407b-8f3d-455dd2a364dc None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.048054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b96cf10-27e7-407b-8f3d-455dd2a364dc None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.081205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa95e82-021d-4422-b733-fdca916d39f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.081875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa95e82-021d-4422-b733-fdca916d39f1 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.082160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa95e82-021d-4422-b733-fdca916d39f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.082437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa95e82-021d-4422-b733-fdca916d39f1 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.083142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa95e82-021d-4422-b733-fdca916d39f1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:39.083453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa95e82-021d-4422-b733-fdca916d39f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.084862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa95e82-021d-4422-b733-fdca916d39f1 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.094824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 486/969] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:39.096090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.131295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e8cb4-01ea-4555-90f0-7792f5efc81b None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.131625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e8cb4-01ea-4555-90f0-7792f5efc81b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.131864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4e8cb4-01ea-4555-90f0-7792f5efc81b None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.133909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 487/970] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:39.138212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88ebfe0-dfc1-4170-b495-8c36870f1afd None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.138507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88ebfe0-dfc1-4170-b495-8c36870f1afd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.138786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88ebfe0-dfc1-4170-b495-8c36870f1afd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b88ebfe0-dfc1-4170-b495-8c36870f1afd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:39.157461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b88ebfe0-dfc1-4170-b495-8c36870f1afd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:39.194788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:39.214385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea8d3fe-5fb2-456b-a0b0-11723f7c382e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.231753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 488/971] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:39 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:39.254979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.255746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.256097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.256246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88ebfe0-dfc1-4170-b495-8c36870f1afd None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.256549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.257326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:39.257703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.259149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.264421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 484/972] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:39.270165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.297458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:39.366058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88ebfe0-dfc1-4170-b495-8c36870f1afd None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.375936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a39240b-fbf3-4471-adde-45ec2b66592a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.402139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.402981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.403267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.403784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.411539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.412201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.412514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.412803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.413605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] RBAC: Adding query filter params (interface=admin, service_id=f0c73fe06e9f46c18267d7b9de0a9d77, region_id=RegionOne) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:39.413881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:39.413983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.415281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.421282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 485/973] 10.222.0.20 () {56 vars in 1249 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/endpoints?service_id=f0c73fe06e9f46c18267d7b9de0a9d77&interface=admin®ion_id=RegionOne => generated 193 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:39.422639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:39.423562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:39.424443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:39.426595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.428755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 489/974] 10.222.0.20 () {56 vars in 1120 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:39.429463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.460707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:39.531611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2b64b8-885c-4d42-8aeb-bba92bc8cc39 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.532932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:39.549499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a869afbf-47f9-4a98-99b1-b7ce1563b33f None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.565521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa48471-7403-4419-a5ed-42a09a484207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.566186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa48471-7403-4419-a5ed-42a09a484207 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.566477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa48471-7403-4419-a5ed-42a09a484207 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.566757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa48471-7403-4419-a5ed-42a09a484207 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.567501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa48471-7403-4419-a5ed-42a09a484207 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:39.567811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa48471-7403-4419-a5ed-42a09a484207 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.569221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa48471-7403-4419-a5ed-42a09a484207 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.577646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 490/975] 10.222.0.20 () {56 vars in 1115 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:39.578854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca796f4-9e43-407c-8b0e-dfd0412226a8 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.579147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca796f4-9e43-407c-8b0e-dfd0412226a8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.579431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca796f4-9e43-407c-8b0e-dfd0412226a8 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.580702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 491/976] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:39.582949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca796f4-9e43-407c-8b0e-dfd0412226a8 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.589381 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf89049-04d0-4255-b7b1-2a5393a39485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.590055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf89049-04d0-4255-b7b1-2a5393a39485 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.590907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf89049-04d0-4255-b7b1-2a5393a39485 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.591248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf89049-04d0-4255-b7b1-2a5393a39485 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.591940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf89049-04d0-4255-b7b1-2a5393a39485 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.593765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf89049-04d0-4255-b7b1-2a5393a39485 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.600550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 486/977] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/services => generated 1497 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 09:58:39.601703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ad8e5b-8d42-40e5-b640-af30f6972896 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.601994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ad8e5b-8d42-40e5-b640-af30f6972896 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.602274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ad8e5b-8d42-40e5-b640-af30f6972896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.624591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-38ad8e5b-8d42-40e5-b640-af30f6972896 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:39.625784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-38ad8e5b-8d42-40e5-b640-af30f6972896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:39.702638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:39.708376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 487/978] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:39 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:39.714532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca796f4-9e43-407c-8b0e-dfd0412226a8 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.730977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ad8e5b-8d42-40e5-b640-af30f6972896 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.753365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.754158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.754447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.754719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.767946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.769698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:39.788753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:39.789917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:39.791014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:39.791014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:39.791014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:39.791014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:39.791014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 492/979] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:39.797598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:39.835334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:39.857618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ad8e5b-8d42-40e5-b640-af30f6972896 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.895554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.896360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.896766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.897140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.897956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.899417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.901009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7058e2b-9288-4bb7-a6f2-ff1971392b4b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:39.913951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 488/980] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:39 2022] POST /identity/v3/services => generated 242 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:58:39.947498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:39.948332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:39.948902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:39.948902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:39.949773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:39.950154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:39.951668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:39.962066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:39.971343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 493/981] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:39.972790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:40.090084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8887f2c2-f7e5-4d68-be06-f14982b857bd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:40.127074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:40.127888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:40.128175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:40.128506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:40.136505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:40.138511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:40.141515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:40.156228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:40.157414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:40.158672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 489/982] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 09:58:39 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:40.164197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:40.265570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81391e9c-fbc2-4f9f-99fc-04453dd5f00a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:40.298979 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:40.299647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:40.299927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:40.300199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:40.301118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:40.301429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:40.302850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:40.320781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 494/983] 10.222.0.20 () {56 vars in 1149 bytes} [Tue Jul 5 09:58:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:40.326171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:40.433021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e2e001-6559-4127-8b3b-21a523ff01bc None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:40.466390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:40.467188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:40.467586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:40.467959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:40.533452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:40.537099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:40.538904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:40.643409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 490/984] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:40 2022] GET /identity/v3/auth/tokens => generated 1560 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:58:40.645037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:40.709983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:40.754621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8138af4-5109-4031-9ccb-cdf9206ae71e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:40.788187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:40.788933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:40.789217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:40.789491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:40.800014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:40.801490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:40.818992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:40.819866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:40.821431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:40.821431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:40.821431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:40.821431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:40.821431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 495/985] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:40.825725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:40.928303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a2bc15-f932-4f52-9f91-91d5b0300921 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:40.930256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:40.964371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:40.965074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:40.965873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:40.966146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:40.966858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:40.967161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:40.968554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:40.980831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 491/986] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:40.985778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:41.088954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25142f79-69e3-4b14-b7bc-ac4262717d00 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:41.100137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:41.124844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:41.125632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:41.125916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:41.126194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:41.132771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:41.134217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:41.147903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:41.148814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:41.149710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 496/987] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:41.156422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:41.255559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3b4b77-dbf4-47f8-bf86-d7dcc54795cb None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:41.288734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a0000e-12cd-4741-8f62-4a2a003d7838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:41.289394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0000e-12cd-4741-8f62-4a2a003d7838 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:41.289675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0000e-12cd-4741-8f62-4a2a003d7838 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:41.289949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0000e-12cd-4741-8f62-4a2a003d7838 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:41.290724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0000e-12cd-4741-8f62-4a2a003d7838 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:41.291032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0000e-12cd-4741-8f62-4a2a003d7838 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:41.292472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0000e-12cd-4741-8f62-4a2a003d7838 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:41.305041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 492/988] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:41.310731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:41.410113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf60676-7b71-4920-b450-213ab7cc20ad None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:41.444175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc1ac1c-ae81-4f70-8166-9fda0fc7b1b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:41.444867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc1ac1c-ae81-4f70-8166-9fda0fc7b1b4 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:41.445228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc1ac1c-ae81-4f70-8166-9fda0fc7b1b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:41.445513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc1ac1c-ae81-4f70-8166-9fda0fc7b1b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:41.446344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc1ac1c-ae81-4f70-8166-9fda0fc7b1b4 None admin] RBAC: Adding query filter params (role.id=5d677ad38b8c41579a9b215dadf5562e, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=154e830e318245158269c03fd2d9621a) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:41.446659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc1ac1c-ae81-4f70-8166-9fda0fc7b1b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:41.448094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc1ac1c-ae81-4f70-8166-9fda0fc7b1b4 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:41.454227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 497/989] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:41 2022] GET /identity/v3/role_assignments?user.id=154e830e318245158269c03fd2d9621a&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=5d677ad38b8c41579a9b215dadf5562e => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:41.648606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ac961-766d-45bc-97ad-60ac3d4e8bfa None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:41.649106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ac961-766d-45bc-97ad-60ac3d4e8bfa None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:41.649193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ac961-766d-45bc-97ad-60ac3d4e8bfa None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:41.650670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 493/990] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:41.657021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43e564a-003c-436e-958a-d8f16f3d1f17 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:41.657314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43e564a-003c-436e-958a-d8f16f3d1f17 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:41.657591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43e564a-003c-436e-958a-d8f16f3d1f17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:41.674929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b43e564a-003c-436e-958a-d8f16f3d1f17 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:41.676033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b43e564a-003c-436e-958a-d8f16f3d1f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:41.704115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:41.745273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 498/991] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:41 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:41.768184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ac961-766d-45bc-97ad-60ac3d4e8bfa None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:41.791551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:41.876485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ac961-766d-45bc-97ad-60ac3d4e8bfa None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:41.912048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5891d7-0d16-4a5f-b516-d5d7c3ac1025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:41.912881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5891d7-0d16-4a5f-b516-d5d7c3ac1025 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:41.913213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5891d7-0d16-4a5f-b516-d5d7c3ac1025 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:41.913492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5891d7-0d16-4a5f-b516-d5d7c3ac1025 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:41.914143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5891d7-0d16-4a5f-b516-d5d7c3ac1025 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:41.915566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5891d7-0d16-4a5f-b516-d5d7c3ac1025 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-8a5891d7-0d16-4a5f-b516-d5d7c3ac1025 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:41.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:41.928950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 05 09:58:41.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:41.932699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 494/992] 10.222.0.20 () {56 vars in 1126 bytes} [Tue Jul 5 09:58:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:41.935635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43e564a-003c-436e-958a-d8f16f3d1f17 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:41.998963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66b381-6277-4fa6-96eb-a21b3e8a4566 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:41.999311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66b381-6277-4fa6-96eb-a21b3e8a4566 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:41.999531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66b381-6277-4fa6-96eb-a21b3e8a4566 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.000941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 495/993] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:42.005654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d3198e-2ccb-4a9b-ba1a-83452ebe3443 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.005947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d3198e-2ccb-4a9b-ba1a-83452ebe3443 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.006230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d3198e-2ccb-4a9b-ba1a-83452ebe3443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.022980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-58d3198e-2ccb-4a9b-ba1a-83452ebe3443 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:42.024156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-58d3198e-2ccb-4a9b-ba1a-83452ebe3443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:42.040215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43e564a-003c-436e-958a-d8f16f3d1f17 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.078738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.079762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.080051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.080332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.081061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:42.081370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.083165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.088332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 499/994] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:42.094192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:42.096851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 496/995] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:42 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:42.118879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d3198e-2ccb-4a9b-ba1a-83452ebe3443 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:42.123243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:42.135520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:42.203471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c62230-92ad-4f34-88bf-177d9d4b8ac5 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.226545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d3198e-2ccb-4a9b-ba1a-83452ebe3443 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.238738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.239407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.239692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.239968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.240715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:42.241022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.242391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.251867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 500/996] 10.222.0.20 () {56 vars in 1122 bytes} [Tue Jul 5 09:58:42 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:42.252736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:42.265176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.265975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.266262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.266541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.267192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.268579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:42.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:42.282143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:42.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 497/997] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:42 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:42.287982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:42.351680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:42.361796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e705ebb3-ddb7-4b36-b653-48b98f7aed48 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.370425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:42.392718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c031d219-9a77-494c-9cc6-e84dfd58ced1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.400579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.401242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.401528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.401934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.403199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.406194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.420980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 501/998] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:42 2022] POST /identity/v3/endpoints => generated 325 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:42.450200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.450882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.451162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.451432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.452153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] RBAC: Adding query filter params (name=compute) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:42.452498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.453984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.459966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 498/999] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:42 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:42.465591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:42.578609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf3e00-f198-4fae-9dac-1834aa44f46e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.617410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.618078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.618360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.618642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.619355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] RBAC: Adding query filter params (type=compute) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:42.619663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.621068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.626786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 502/1000] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:42 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:42.632768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:42.736592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e909cbaf-2376-4e6c-a65b-a26c9c1be50c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.770524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5964392-8917-41ad-91fd-cf64720a38c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.771184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5964392-8917-41ad-91fd-cf64720a38c5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.771464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5964392-8917-41ad-91fd-cf64720a38c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.771735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5964392-8917-41ad-91fd-cf64720a38c5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.772557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5964392-8917-41ad-91fd-cf64720a38c5 None admin] RBAC: Adding query filter params (interface=public, service_id=d55c20bd6ff84c9494fcd6880a46331a, region_id=RegionOne) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:42.772868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5964392-8917-41ad-91fd-cf64720a38c5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.774236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5964392-8917-41ad-91fd-cf64720a38c5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.779409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 499/1001] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 09:58:42 2022] GET /identity/v3/endpoints?service_id=d55c20bd6ff84c9494fcd6880a46331a&interface=public®ion_id=RegionOne => generated 194 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:42.784692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:42.802812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:42.886369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a7d5d-be25-4f95-8ab3-8a0e9aba70ab None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:42.924707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b993a67-2706-4f71-be64-f9b33ba42c7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:42.924707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b993a67-2706-4f71-be64-f9b33ba42c7d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:42.924707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b993a67-2706-4f71-be64-f9b33ba42c7d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:42.924707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b993a67-2706-4f71-be64-f9b33ba42c7d None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:42.925109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b993a67-2706-4f71-be64-f9b33ba42c7d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:42.927356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b993a67-2706-4f71-be64-f9b33ba42c7d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:42.935707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 503/1002] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:42 2022] GET /identity/v3/services => generated 1727 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 09:58:43.542235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd91f14-43a2-40ae-8bce-38fd69a69dd1 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:43.542645 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd91f14-43a2-40ae-8bce-38fd69a69dd1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:43.542821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd91f14-43a2-40ae-8bce-38fd69a69dd1 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:43.545087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 500/1003] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:43.550821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac59e243-21df-49be-8367-6712f678a9df None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:43.551346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac59e243-21df-49be-8367-6712f678a9df None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:43.551346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac59e243-21df-49be-8367-6712f678a9df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:43.568775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ac59e243-21df-49be-8367-6712f678a9df None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:43.569963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ac59e243-21df-49be-8367-6712f678a9df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:43.639338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 504/1004] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:43 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:43.661960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd91f14-43a2-40ae-8bce-38fd69a69dd1 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:43.664892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:43.764944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd91f14-43a2-40ae-8bce-38fd69a69dd1 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:43.798655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:43.799380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:43.799663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:43.799940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] PATH_INFO: `/v3/services/image` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:43.800617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:43.801989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:43.814047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:43.815201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:43.816061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 501/1005] 10.222.0.20 () {56 vars in 1105 bytes} [Tue Jul 5 09:58:43 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:43.821209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac59e243-21df-49be-8367-6712f678a9df None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:43.832876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:43.919482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac59e243-21df-49be-8367-6712f678a9df None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:43.960020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45b5e45-42b2-4847-bcd3-cb99669d48d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:43.961052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45b5e45-42b2-4847-bcd3-cb99669d48d6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:43.961533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45b5e45-42b2-4847-bcd3-cb99669d48d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:43.962003 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45b5e45-42b2-4847-bcd3-cb99669d48d6 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:43.962915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45b5e45-42b2-4847-bcd3-cb99669d48d6 None admin] RBAC: Adding query filter params (name=image) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:43.963430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45b5e45-42b2-4847-bcd3-cb99669d48d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:43.965016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45b5e45-42b2-4847-bcd3-cb99669d48d6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:43.971017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 505/1006] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:43 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:43.976411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:44.044432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:44.081886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba46e3-a17b-4033-a2b9-84a5dec893d5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:44.115443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0e6a39-d976-44cc-9808-b07f9a60094d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:44.116113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e6a39-d976-44cc-9808-b07f9a60094d None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:44.116421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e6a39-d976-44cc-9808-b07f9a60094d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:44.116701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e6a39-d976-44cc-9808-b07f9a60094d None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:44.117403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0e6a39-d976-44cc-9808-b07f9a60094d None admin] RBAC: Adding query filter params (type=image) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:44.117711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0e6a39-d976-44cc-9808-b07f9a60094d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:44.119083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0e6a39-d976-44cc-9808-b07f9a60094d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:44.130476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 502/1007] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:43 2022] GET /identity/v3/services?type=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:44.469143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fe1f2b-39b8-4006-ae8b-bfe2815aa561 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:44.469538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fe1f2b-39b8-4006-ae8b-bfe2815aa561 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:44.469720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fe1f2b-39b8-4006-ae8b-bfe2815aa561 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:44.471238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 506/1008] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:44.476149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb46db11-4680-4e72-97a2-62c79880a4fa None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:44.476475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb46db11-4680-4e72-97a2-62c79880a4fa None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:44.476757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb46db11-4680-4e72-97a2-62c79880a4fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:44.494330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-bb46db11-4680-4e72-97a2-62c79880a4fa None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:44.495506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bb46db11-4680-4e72-97a2-62c79880a4fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:44.573310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 503/1009] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:44 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:44.597670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fe1f2b-39b8-4006-ae8b-bfe2815aa561 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:44.674923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:44.701910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fe1f2b-39b8-4006-ae8b-bfe2815aa561 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:44.735953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7244326-9106-446d-a524-62d01462a842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:44.736674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7244326-9106-446d-a524-62d01462a842 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:44.736960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7244326-9106-446d-a524-62d01462a842 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:44.737234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7244326-9106-446d-a524-62d01462a842 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:44.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7244326-9106-446d-a524-62d01462a842 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:44.739816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7244326-9106-446d-a524-62d01462a842 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:44.746096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 507/1010] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:44.747874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb46db11-4680-4e72-97a2-62c79880a4fa None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:44.810016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:44.853233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb46db11-4680-4e72-97a2-62c79880a4fa None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:44.888461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe61f86d-48b4-4422-bfb1-02e604b0ffa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:44.889403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe61f86d-48b4-4422-bfb1-02e604b0ffa1 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:44.889912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe61f86d-48b4-4422-bfb1-02e604b0ffa1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:44.890390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe61f86d-48b4-4422-bfb1-02e604b0ffa1 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:44.891277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe61f86d-48b4-4422-bfb1-02e604b0ffa1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:44.892926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe61f86d-48b4-4422-bfb1-02e604b0ffa1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:44.919702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 504/1011] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:44 2022] POST /identity/v3/projects => generated 311 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:44.969346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955e0b55-7e47-4d48-896d-71df8700d933 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:44.969652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955e0b55-7e47-4d48-896d-71df8700d933 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:44.969906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955e0b55-7e47-4d48-896d-71df8700d933 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:44.971368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 508/1012] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:44.977456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed80af6-ed6e-4e6f-b79e-e0794472bda6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:44.977788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed80af6-ed6e-4e6f-b79e-e0794472bda6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:44.978035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed80af6-ed6e-4e6f-b79e-e0794472bda6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:44.996471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-fed80af6-ed6e-4e6f-b79e-e0794472bda6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:44.997632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fed80af6-ed6e-4e6f-b79e-e0794472bda6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:45.072870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 505/1013] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:44 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:45.096090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955e0b55-7e47-4d48-896d-71df8700d933 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:45.202819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955e0b55-7e47-4d48-896d-71df8700d933 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:45.239998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:45.240880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:45.241165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:45.241435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:45.242078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:45.243446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:45.255189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:45.256171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:45.257061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 509/1014] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:45 2022] GET /identity/v3/services/compute => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:45.264765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed80af6-ed6e-4e6f-b79e-e0794472bda6 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:45.308273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:45.365976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed80af6-ed6e-4e6f-b79e-e0794472bda6 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:45.400612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:45.401276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:45.401555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:45.401829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:45.402529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] RBAC: Adding query filter params (name=compute) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:45.403391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:45.404799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:45.414028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 506/1015] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:45 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:45.415273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:45.426425 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:45.519447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293a35e6-0ca1-486c-a337-b47ef4dbb534 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:45.552818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c4bcc1-c652-42ad-85db-f9b5b0afd2e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:45.553476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c4bcc1-c652-42ad-85db-f9b5b0afd2e6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:45.553757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c4bcc1-c652-42ad-85db-f9b5b0afd2e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:45.554029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c4bcc1-c652-42ad-85db-f9b5b0afd2e6 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:45.554729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c4bcc1-c652-42ad-85db-f9b5b0afd2e6 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:45.555036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c4bcc1-c652-42ad-85db-f9b5b0afd2e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:45.556412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c4bcc1-c652-42ad-85db-f9b5b0afd2e6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:45.566376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 510/1016] 10.222.0.20 () {56 vars in 1112 bytes} [Tue Jul 5 09:58:45 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:45.567612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:45.672767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ea7a2-f2c3-4a42-9c53-038efc4ef25c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:45.680438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:45.708728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4231b87-009d-4b03-9ba4-733d8f5789fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:45.709402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4231b87-009d-4b03-9ba4-733d8f5789fc None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:45.709690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4231b87-009d-4b03-9ba4-733d8f5789fc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:45.709971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4231b87-009d-4b03-9ba4-733d8f5789fc None admin] PATH_INFO: `/v3/endpoints` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:45.710721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4231b87-009d-4b03-9ba4-733d8f5789fc None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:45.712098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4231b87-009d-4b03-9ba4-733d8f5789fc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:45.732620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 507/1017] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:45 2022] POST /identity/v3/endpoints => generated 334 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:58:46.165650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb2a8fb-fd6d-4fec-8335-bc882dea2a73 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:46.166159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb2a8fb-fd6d-4fec-8335-bc882dea2a73 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:46.166268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb2a8fb-fd6d-4fec-8335-bc882dea2a73 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:46.167715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 511/1018] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:46.173262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa32e82-8d5c-49a2-abe6-b2783e273c75 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:46.173634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa32e82-8d5c-49a2-abe6-b2783e273c75 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:46.173819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa32e82-8d5c-49a2-abe6-b2783e273c75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:46.191049 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6fa32e82-8d5c-49a2-abe6-b2783e273c75 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:46.192155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6fa32e82-8d5c-49a2-abe6-b2783e273c75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:46.263883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 508/1019] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:46 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:46.286051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb2a8fb-fd6d-4fec-8335-bc882dea2a73 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:46.358111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:46.390333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb2a8fb-fd6d-4fec-8335-bc882dea2a73 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:46.424015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03f4108-fe1c-4744-bc71-ca7c9f2ad03d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:46.424701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03f4108-fe1c-4744-bc71-ca7c9f2ad03d None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:46.424989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03f4108-fe1c-4744-bc71-ca7c9f2ad03d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:46.425265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03f4108-fe1c-4744-bc71-ca7c9f2ad03d None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:46.425985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03f4108-fe1c-4744-bc71-ca7c9f2ad03d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:46.427341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b03f4108-fe1c-4744-bc71-ca7c9f2ad03d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:46.443571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 512/1020] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:58:46 2022] POST /identity/v3/services => generated 242 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:47.006179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa8f9fb-2563-4aac-b2b9-a00d6079f98a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:47.006545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa8f9fb-2563-4aac-b2b9-a00d6079f98a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:47.006750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa8f9fb-2563-4aac-b2b9-a00d6079f98a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:47.008182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 509/1021] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:47.013964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71338e51-445f-4dde-8bf6-fa870b93f043 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:47.014261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71338e51-445f-4dde-8bf6-fa870b93f043 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:47.014544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71338e51-445f-4dde-8bf6-fa870b93f043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:47.032145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-71338e51-445f-4dde-8bf6-fa870b93f043 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:47.033372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-71338e51-445f-4dde-8bf6-fa870b93f043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:47.107456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 513/1022] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:47 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:47.131637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa8f9fb-2563-4aac-b2b9-a00d6079f98a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:47.222460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:47.238000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa8f9fb-2563-4aac-b2b9-a00d6079f98a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:47.272440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7b6341-906f-405a-8705-9597052ff6f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:47.273234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7b6341-906f-405a-8705-9597052ff6f4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:47.273524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7b6341-906f-405a-8705-9597052ff6f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:47.273806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7b6341-906f-405a-8705-9597052ff6f4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:47.274988 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7b6341-906f-405a-8705-9597052ff6f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:47.276452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7b6341-906f-405a-8705-9597052ff6f4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:47.278343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 510/1023] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:47.283681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71338e51-445f-4dde-8bf6-fa870b93f043 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:47.339101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:47.384878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71338e51-445f-4dde-8bf6-fa870b93f043 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:47.419462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8100af1-834c-4cfe-815b-56423f805a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:47.420140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8100af1-834c-4cfe-815b-56423f805a3f None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:47.420451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8100af1-834c-4cfe-815b-56423f805a3f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:47.420735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8100af1-834c-4cfe-815b-56423f805a3f None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:47.421456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8100af1-834c-4cfe-815b-56423f805a3f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:47.422888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8100af1-834c-4cfe-815b-56423f805a3f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:47.450136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 514/1024] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:58:47 2022] POST /identity/v3/users => generated 285 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:58:47.861146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534c350d-00cb-440f-82b2-37ddcf25ee14 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:47.861475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534c350d-00cb-440f-82b2-37ddcf25ee14 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:47.861704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534c350d-00cb-440f-82b2-37ddcf25ee14 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:47.863214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 511/1025] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:47.868080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7603cb0-302f-4304-acd6-5fa6c598179f None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:47.868376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7603cb0-302f-4304-acd6-5fa6c598179f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:47.868681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7603cb0-302f-4304-acd6-5fa6c598179f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:47.885625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f7603cb0-302f-4304-acd6-5fa6c598179f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:47.886810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f7603cb0-302f-4304-acd6-5fa6c598179f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:47.962621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 515/1026] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:47 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:47.987023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534c350d-00cb-440f-82b2-37ddcf25ee14 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.089110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534c350d-00cb-440f-82b2-37ddcf25ee14 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.123637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.124370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.124727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.125007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.135135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:48.143391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:48.144313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:48.145465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:48.148558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 512/1027] 10.222.0.20 () {56 vars in 1120 bytes} [Tue Jul 5 09:58:47 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:48.150995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7603cb0-302f-4304-acd6-5fa6c598179f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.194391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:48.252355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7603cb0-302f-4304-acd6-5fa6c598179f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.287272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda59d55-14f7-4a2b-b08e-b6fc6b6bef02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.287964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda59d55-14f7-4a2b-b08e-b6fc6b6bef02 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.288206 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda59d55-14f7-4a2b-b08e-b6fc6b6bef02 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.288501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda59d55-14f7-4a2b-b08e-b6fc6b6bef02 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.289240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda59d55-14f7-4a2b-b08e-b6fc6b6bef02 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:48.289542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda59d55-14f7-4a2b-b08e-b6fc6b6bef02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:48.290898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda59d55-14f7-4a2b-b08e-b6fc6b6bef02 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:48.299898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 516/1028] 10.222.0.20 () {56 vars in 1115 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:48.304346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.323290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:48.407738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb21e6ad-de31-41bc-a055-7dc6c646177a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.441355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.442426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.442885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.443332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.456013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:48.457698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:48.470058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0c050b-0649-4b05-8e91-144705e0d6aa None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.470346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0c050b-0649-4b05-8e91-144705e0d6aa None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.470626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0c050b-0649-4b05-8e91-144705e0d6aa None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.472005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 517/1029] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:48.476374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:48.477267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:48.478146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:48.478146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:48.478146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:48.478146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:48.478363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568e2cbb-a3f8-4f4a-993d-40496570132d None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.478363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568e2cbb-a3f8-4f4a-993d-40496570132d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.478363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568e2cbb-a3f8-4f4a-993d-40496570132d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.480565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 513/1030] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:48.483035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.495303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-568e2cbb-a3f8-4f4a-993d-40496570132d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:48.496777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-568e2cbb-a3f8-4f4a-993d-40496570132d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:48.552595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:48.572435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 518/1031] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:48 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:48.592340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bd9c24-7769-4542-9716-1783dbc1021a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.592742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568e2cbb-a3f8-4f4a-993d-40496570132d None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.630939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.631617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.631907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.632190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.632948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:48.633257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:48.634633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:48.636189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:48.653350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 514/1032] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:48.654232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.707421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568e2cbb-a3f8-4f4a-993d-40496570132d None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.744543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.745332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.745611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.745883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] PATH_INFO: `/v3/services/image` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.746558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:48.748036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:48.759123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a17d6e9-98ee-4aaa-847d-b7b385fed741 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:48.760340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:48.761324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:48.762217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 519/1033] 10.222.0.20 () {56 vars in 1105 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/services/image => generated 86 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:48.767431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.790665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:48.799335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.800137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.800431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.800717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.807676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:48.809178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:48.824504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:48.825639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:48.826772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:48.830368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 515/1034] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:48.831131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.873061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:48.878350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b53dab-3879-433a-bb6e-6425cef170d6 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.913620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.914292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.914579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.914863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.915582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] RBAC: Adding query filter params (name=image) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:48.915874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:48.917280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:48.927233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 520/1035] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:48.929107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:48.942720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8218f1ac-7313-4c5f-a52a-f79b1b2a6c0e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:48.978094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:48.978880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:48.978991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:48.979240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:48.980076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:48.980385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:48.982457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.000274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 516/1036] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:49.005680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.035751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fd9d1b-b4d6-4352-84af-d6e4310f0941 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.056234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:49.070955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.071728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.072020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.072304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.073064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] RBAC: Adding query filter params (type=image) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:49.073376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.075282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.085033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 521/1037] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:48 2022] GET /identity/v3/services?type=image => generated 350 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:49.086229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.114035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:49.121936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fa206c-1ea4-4854-9c92-52cc72215e95 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.158852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.159505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.159790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.160065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.191278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37dadf2-9148-4607-84fd-5bfc12c46937 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.230504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.231457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.231925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.232128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.232437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.232718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.233516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] RBAC: Adding query filter params (interface=public, service_id=29b6432c17c34a74963e9885cbcac8f5, region_id=RegionOne) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:49.233833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.235201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.240837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 522/1038] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/endpoints?service_id=29b6432c17c34a74963e9885cbcac8f5&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:49.245840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.283264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:49.345380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 517/1039] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:58:49.346451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.355393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:49.364523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7752f-7948-4dfa-bf47-1291569281ec None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.400297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4750e71a-037f-42a9-8eb9-a820881a80f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.401227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4750e71a-037f-42a9-8eb9-a820881a80f0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.401508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4750e71a-037f-42a9-8eb9-a820881a80f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.401783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4750e71a-037f-42a9-8eb9-a820881a80f0 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.402470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4750e71a-037f-42a9-8eb9-a820881a80f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.404381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4750e71a-037f-42a9-8eb9-a820881a80f0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.411884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 523/1040] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/services => generated 1957 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 09:58:49.464171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0563491-421e-4b5e-8574-17d9bf9fe6b8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.502697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.503429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.503713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.503992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.516230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.517747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.525738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:49.539394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:49.540410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:49.541338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:49.541338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:49.541338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:49.541338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:49.541338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 518/1041] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:49.541651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b6ec8-28e9-4c1b-9f2a-004d92dcc0fd None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.541651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b6ec8-28e9-4c1b-9f2a-004d92dcc0fd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.541651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b6ec8-28e9-4c1b-9f2a-004d92dcc0fd None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.548436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 524/1042] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:49.548632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.548940 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a54a20-39da-4e2d-aeb8-f03a3b1dfac1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.549236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a54a20-39da-4e2d-aeb8-f03a3b1dfac1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.549519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a54a20-39da-4e2d-aeb8-f03a3b1dfac1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.566480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-36a54a20-39da-4e2d-aeb8-f03a3b1dfac1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:49.567761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-36a54a20-39da-4e2d-aeb8-f03a3b1dfac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:49.644215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 525/1043] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:49 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:49.661020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a21554-9243-49ef-bf2a-565a96d47205 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.665416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a54a20-39da-4e2d-aeb8-f03a3b1dfac1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.696986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.697655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.697941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.698220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.698938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:49.699245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.700795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.712517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 519/1044] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:49.718345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.754548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:49.771428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a54a20-39da-4e2d-aeb8-f03a3b1dfac1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.781083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:49.807067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.807863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.808147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.808654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.818656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:49.826827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:49.827759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:49.828658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:49.829334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623dcec3-6a8d-44d4-8674-48abaa444154 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.833016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 526/1045] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:49.833796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.863481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.864275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.864590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.864876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.872619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.874113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:49.888789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:49.889894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:49.890813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 520/1046] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:49.895798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:49.949519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27a9135-f076-47a9-85dd-8c5318bb7a57 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:49.987670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:49.988345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:49.988658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:49.988939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:49.989810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:49.990121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:49.991492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:50.005376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 527/1047] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:50.005509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d011112e-5496-43dd-92d5-e2e5833205c3 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:50.006290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:50.015375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:50.024161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:50.042242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:50.042914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:50.043198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:50.044008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:50.044829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:50.045143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:50.046583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:50.058496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 521/1048] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:50.064546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:50.114946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447c5f1-2141-40d0-acc4-92f01e2a8937 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:50.151314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:50.152476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:50.152942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:50.153367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] PATH_INFO: `/v3/users/9a6e4c3d068b48d883f4b20d88846c19` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:50.167656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] RBAC: Authorizing `identity:get_user(user_id=9a6e4c3d068b48d883f4b20d88846c19)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:50.169377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:50.169499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaa398d-b526-47a1-81e6-551407ec92c7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:50.171843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 528/1049] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:58:50 2022] GET /identity/v3/users/9a6e4c3d068b48d883f4b20d88846c19 => generated 285 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:50.176986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:50.209455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:50.210124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:50.210405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:50.210682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:50.211509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] RBAC: Adding query filter params (role.id=5d677ad38b8c41579a9b215dadf5562e, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=f99a9fe4b40646d389c3ebf47358956c) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:50.211817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:50.213260 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:50.218637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 522/1050] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:50 2022] GET /identity/v3/role_assignments?user.id=f99a9fe4b40646d389c3ebf47358956c&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=5d677ad38b8c41579a9b215dadf5562e => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:50.243429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:50.287486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce86caf-7f5e-42da-a40d-90c19ccca552 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:50.324101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:50.324868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:50.325154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:50.325431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:50.397000 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:50.398455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:50.458278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:50.503193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 529/1051] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:50 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:58:50.504664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:50.551836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:50.611415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c7fffc-70a8-4115-b5e2-d9d52a1538f1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:50.646454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bb440e-1f22-4834-9cc1-aa39666c9be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:50.647444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb440e-1f22-4834-9cc1-aa39666c9be3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:50.647829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb440e-1f22-4834-9cc1-aa39666c9be3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:50.648203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bb440e-1f22-4834-9cc1-aa39666c9be3 None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:50.649507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bb440e-1f22-4834-9cc1-aa39666c9be3 None admin] RBAC: Authorizing `identity:get_project(project_id=bfdaee025af14785b5f72b76b0d2786b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:50.651046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bb440e-1f22-4834-9cc1-aa39666c9be3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:50.658910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 523/1052] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:58:50 2022] GET /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:50.659894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:50.758869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108ed4da-886b-4f55-94b7-070fc73deef0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:50.792945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f769081-0b76-4607-b772-226e8bfc05ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:50.793623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f769081-0b76-4607-b772-226e8bfc05ad None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:50.793910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f769081-0b76-4607-b772-226e8bfc05ad None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:50.794190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f769081-0b76-4607-b772-226e8bfc05ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:50.795015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f769081-0b76-4607-b772-226e8bfc05ad None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=bfdaee025af14785b5f72b76b0d2786b, user.id=9a6e4c3d068b48d883f4b20d88846c19) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:50.795329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f769081-0b76-4607-b772-226e8bfc05ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:50.796770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f769081-0b76-4607-b772-226e8bfc05ad None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:50.808735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 530/1053] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:50 2022] GET /identity/v3/role_assignments?user.id=9a6e4c3d068b48d883f4b20d88846c19&scope.project.id=bfdaee025af14785b5f72b76b0d2786b&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:51.463732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bb1f7-1ec5-459e-886b-5f0386b6adca None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:51.464062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bb1f7-1ec5-459e-886b-5f0386b6adca None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:51.464292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82bb1f7-1ec5-459e-886b-5f0386b6adca None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:51.465777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 524/1054] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:51.470928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1f5cb4-8db0-450e-997f-e20a343f142c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:51.471394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1f5cb4-8db0-450e-997f-e20a343f142c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:51.471537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1f5cb4-8db0-450e-997f-e20a343f142c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:51.489055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-cf1f5cb4-8db0-450e-997f-e20a343f142c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:51.490241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cf1f5cb4-8db0-450e-997f-e20a343f142c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:51.513388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:51.558270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 531/1055] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:51 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:58:51.580982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bb1f7-1ec5-459e-886b-5f0386b6adca None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:51.685103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82bb1f7-1ec5-459e-886b-5f0386b6adca None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:51.695835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:51.720282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:51.721037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:51.721324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:51.721597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] PATH_INFO: `/v3/services/image` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:51.722278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:51.723615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:51.735654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:51.736652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 05 09:58:51.737532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:51.740718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 525/1056] 10.222.0.20 () {56 vars in 1105 bytes} [Tue Jul 5 09:58:51 2022] GET /identity/v3/services/image => generated 86 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:51.743079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1f5cb4-8db0-450e-997f-e20a343f142c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:51.846131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1f5cb4-8db0-450e-997f-e20a343f142c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:51.882253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:51.882928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:51.883303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:51.883721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:51.884474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] RBAC: Adding query filter params (name=image) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:51.884790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:51.886176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:51.896033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 532/1057] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:51 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:51.897623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:52.008020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e46d84-726f-4f22-884b-71e95276e02c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:52.042291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faea43a5-56c0-48b8-9569-e19d6fbdeb77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:52.042968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea43a5-56c0-48b8-9569-e19d6fbdeb77 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.043256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea43a5-56c0-48b8-9569-e19d6fbdeb77 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.044074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea43a5-56c0-48b8-9569-e19d6fbdeb77 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.044875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faea43a5-56c0-48b8-9569-e19d6fbdeb77 None admin] RBAC: Adding query filter params (type=image) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:52.045187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faea43a5-56c0-48b8-9569-e19d6fbdeb77 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:52.046635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faea43a5-56c0-48b8-9569-e19d6fbdeb77 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:52.055993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 526/1058] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:51 2022] GET /identity/v3/services?type=image => generated 350 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:52.057626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:52.083142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:52.160437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dec24c-5cc0-45ee-8b77-09159cf97100 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:52.194670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:52.195332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.195622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.195900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.196660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:52.198033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:52.209907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da60e1db-7ce7-41e3-ae2f-ce9b05f5b78b None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.210561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da60e1db-7ce7-41e3-ae2f-ce9b05f5b78b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.210871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da60e1db-7ce7-41e3-ae2f-ce9b05f5b78b None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.212477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 527/1059] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:52.217436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08028c1f-8df2-49ec-90b0-488c5c3562ea None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.217759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08028c1f-8df2-49ec-90b0-488c5c3562ea None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.218043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08028c1f-8df2-49ec-90b0-488c5c3562ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.223124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 533/1060] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 09:58:52 2022] POST /identity/v3/endpoints => generated 327 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:58:52.236630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-08028c1f-8df2-49ec-90b0-488c5c3562ea None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:52.237870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-08028c1f-8df2-49ec-90b0-488c5c3562ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:52.275096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:52.328753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 528/1061] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:52.349848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:52.437579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:52.461398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ea37d4-2462-4594-8b9a-25da7c94ff72 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:52.497950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:52.498832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.499112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.499385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.509710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:52.517904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:52.518840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:52.519833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 534/1062] 10.222.0.20 () {56 vars in 1120 bytes} [Tue Jul 5 09:58:52 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:52.525823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08028c1f-8df2-49ec-90b0-488c5c3562ea None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:52.628575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08028c1f-8df2-49ec-90b0-488c5c3562ea None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:52.664753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98300131-61d4-4853-b785-59ac471529c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:52.665432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98300131-61d4-4853-b785-59ac471529c0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.665717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98300131-61d4-4853-b785-59ac471529c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.665997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98300131-61d4-4853-b785-59ac471529c0 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.666753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98300131-61d4-4853-b785-59ac471529c0 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:52.667062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98300131-61d4-4853-b785-59ac471529c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:52.668522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98300131-61d4-4853-b785-59ac471529c0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:52.681951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 529/1063] 10.222.0.20 () {56 vars in 1115 bytes} [Tue Jul 5 09:58:52 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:52.683259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:52.785429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d805fe25-a5c0-48a1-b598-eaa4279687bd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:52.800011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693210c6-5f24-44aa-ac2e-866d3f697aa0 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.800303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693210c6-5f24-44aa-ac2e-866d3f697aa0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.800612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693210c6-5f24-44aa-ac2e-866d3f697aa0 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.801996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 530/1064] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:52.806829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb677a-a8e8-460e-b352-434bb90abf07 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.807117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb677a-a8e8-460e-b352-434bb90abf07 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.807398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb677a-a8e8-460e-b352-434bb90abf07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.819898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:52.820723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:52.821006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:52.821282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:52.823740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2beb677a-a8e8-460e-b352-434bb90abf07 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:52.825502 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2beb677a-a8e8-460e-b352-434bb90abf07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:52.828994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:52.835999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:52.837487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:52.848164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:52.855398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:52.856305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:52.857265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:52.857265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:52.857265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:52.857265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:52.857265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 535/1065] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:52.862146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:52.904438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 531/1066] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:52.926370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2beb677a-a8e8-460e-b352-434bb90abf07 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:52.979068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d450f2-3409-4b4b-9162-d767e912a324 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.015715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.016753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.017269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.017755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.018830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:53.019216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.020880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.027428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2beb677a-a8e8-460e-b352-434bb90abf07 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.039113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 536/1067] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:53.040036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.066240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.067050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.067336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.067617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.078084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:53.085717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:53.086833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:53.087944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:53.092421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 532/1068] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:53.093243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.095491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:53.111829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:53.152082 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22feecd-369e-4eb8-b826-3118c8da1617 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.188733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.189520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.189805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.190080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.193904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51543a97-beed-4de8-93f9-d4704d433a36 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.197621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.199078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:53.214233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:53.215891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 537/1069] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:53.221868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.236977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.237741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.238029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.239061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:53.239370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.240768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.249418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 533/1070] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:53.254867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.331280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bc38b2-91ca-4199-b23d-03787ef10740 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.340909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:53.342712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:53.366509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5ab03-3be2-4104-8705-04cd762fd674 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.366834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.367460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.367746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.368023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.368914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:53.369225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.370705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.391853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 538/1071] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:53.397851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.405191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.406074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.406356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.406632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] PATH_INFO: `/v3/users/9a6e4c3d068b48d883f4b20d88846c19` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.407863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] RBAC: Authorizing `identity:get_user(user_id=9a6e4c3d068b48d883f4b20d88846c19)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.409318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.411106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 534/1072] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/users/9a6e4c3d068b48d883f4b20d88846c19 => generated 285 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:53.416530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.508329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1651f-a218-4137-95c7-6be6e68f8f06 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.522838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1328af29-947d-4dbf-bf2d-70bfdebf8ac4 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.544936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.545728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.546008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.546278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.557344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.557800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.558757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.558857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.559087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.559862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.561070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] RBAC: Authorizing `identity:get_project(project_id=bfdaee025af14785b5f72b76b0d2786b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.562947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.564843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 535/1073] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:53.570515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:53.579971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:53.581066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:53.582012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:53.582012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:53.582012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:53.582012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:53.582012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 539/1074] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:53.586851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.587955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:53.609110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:53.678656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962f4d5-b2df-4dda-8f04-fcc34926cd5a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.694085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253be55f-012a-410e-a54a-f42540d07019 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.716933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.717887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.718175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.718473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b/users/9a6e4c3d068b48d883f4b20d88846c19/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.720662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] RBAC: Authorizing `identity:create_grant(project_id=bfdaee025af14785b5f72b76b0d2786b, user_id=9a6e4c3d068b48d883f4b20d88846c19, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.722339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.730048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.730779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.731091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.731383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.732111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:53.732442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.733827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.736659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 536/1075] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:58:53 2022] PUT /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b/users/9a6e4c3d068b48d883f4b20d88846c19/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 09:58:53.747104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 540/1076] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:53.751661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:53.838458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:53.868028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdec56b0-6c71-48de-b5cc-cf55e213d2a0 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:53.908287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:53.909190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:53.909474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:53.909751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:53.916725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:53.918194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:53.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:53.936470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:53.937382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:53.940795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 537/1077] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:53.942471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:54.025183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:54.046521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f9cd75-b6a6-4b84-8da6-79eb02f1cf5a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:54.079210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55fc039-d685-4d61-a754-cad957e5db3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:54.079868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fc039-d685-4d61-a754-cad957e5db3e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:54.080149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fc039-d685-4d61-a754-cad957e5db3e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:54.080453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55fc039-d685-4d61-a754-cad957e5db3e None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:54.081235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fc039-d685-4d61-a754-cad957e5db3e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:54.081546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fc039-d685-4d61-a754-cad957e5db3e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:54.082965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55fc039-d685-4d61-a754-cad957e5db3e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:54.100337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 541/1078] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:54.101911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:54.205375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6c2976-4371-4c7a-a40f-f911546e671c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:54.241346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0f70bd-c670-4c8a-ba6a-8db6f930542d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:54.242236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0f70bd-c670-4c8a-ba6a-8db6f930542d None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:54.242521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0f70bd-c670-4c8a-ba6a-8db6f930542d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:54.242799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0f70bd-c670-4c8a-ba6a-8db6f930542d None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/f99a9fe4b40646d389c3ebf47358956c/roles/5d677ad38b8c41579a9b215dadf5562e` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:54.245116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0f70bd-c670-4c8a-ba6a-8db6f930542d None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=f99a9fe4b40646d389c3ebf47358956c, role_id=5d677ad38b8c41579a9b215dadf5562e)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:54.246713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0f70bd-c670-4c8a-ba6a-8db6f930542d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:54.248934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:54.261570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 538/1079] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:58:54 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/f99a9fe4b40646d389c3ebf47358956c/roles/5d677ad38b8c41579a9b215dadf5562e => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:58:54.312224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a6ce58-8b7d-4081-abff-ccd6ec3a01f1 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:54.312651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a6ce58-8b7d-4081-abff-ccd6ec3a01f1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:54.312993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a6ce58-8b7d-4081-abff-ccd6ec3a01f1 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:54.314814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 542/1080] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:54.320514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce35ce7-4f41-46b3-acca-2866f75fe3bc None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:54.320855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce35ce7-4f41-46b3-acca-2866f75fe3bc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:54.321143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce35ce7-4f41-46b3-acca-2866f75fe3bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:54.339260 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6ce35ce7-4f41-46b3-acca-2866f75fe3bc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:54.341094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6ce35ce7-4f41-46b3-acca-2866f75fe3bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:54.413897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 539/1081] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:54 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:54.436059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a6ce58-8b7d-4081-abff-ccd6ec3a01f1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:54.541898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a6ce58-8b7d-4081-abff-ccd6ec3a01f1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:54.575371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:54.576093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:54.576380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:54.576678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:54.587433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:54.646141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:54.647588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:54.750693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 543/1082] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:54 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:58:54.752573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce35ce7-4f41-46b3-acca-2866f75fe3bc None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:54.854191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce35ce7-4f41-46b3-acca-2866f75fe3bc None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:54.878253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:54.891612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5da1add-f059-46ea-975f-a86565adc94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:54.892436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5da1add-f059-46ea-975f-a86565adc94b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:54.892723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5da1add-f059-46ea-975f-a86565adc94b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:54.892999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5da1add-f059-46ea-975f-a86565adc94b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:54.903847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5da1add-f059-46ea-975f-a86565adc94b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:54.905321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5da1add-f059-46ea-975f-a86565adc94b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-b5da1add-f059-46ea-975f-a86565adc94b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:54.923466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:54.924402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:54.930209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:54.930209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:54.930209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:54.930209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:54.930209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 540/1083] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:54.936196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:54.979184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:55.038495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1307f6de-eeea-4a30-ad39-419676d1dc63 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:55.071015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeef934b-baab-4f7d-a703-3dbbd56631f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:55.071675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeef934b-baab-4f7d-a703-3dbbd56631f0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:55.071961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeef934b-baab-4f7d-a703-3dbbd56631f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:55.072286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeef934b-baab-4f7d-a703-3dbbd56631f0 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:55.073033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeef934b-baab-4f7d-a703-3dbbd56631f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:55.073340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeef934b-baab-4f7d-a703-3dbbd56631f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:55.075226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeef934b-baab-4f7d-a703-3dbbd56631f0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:55.092753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 544/1084] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:55.769430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96385aa7-a8e8-44bf-a2d6-6adb942856fe None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:55.770052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96385aa7-a8e8-44bf-a2d6-6adb942856fe None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:55.770052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96385aa7-a8e8-44bf-a2d6-6adb942856fe None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:55.771608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 541/1085] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:55.776965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927b039-d441-4900-80ff-906c6e468d26 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:55.777363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927b039-d441-4900-80ff-906c6e468d26 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:55.777662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5927b039-d441-4900-80ff-906c6e468d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:55.794879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5927b039-d441-4900-80ff-906c6e468d26 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:55.796058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5927b039-d441-4900-80ff-906c6e468d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:55.860413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 545/1086] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:55 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:55.884131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96385aa7-a8e8-44bf-a2d6-6adb942856fe None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:55.991310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96385aa7-a8e8-44bf-a2d6-6adb942856fe None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.024978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.025701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.025987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.026266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.036252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:56.044499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:56.045410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:56.046591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 542/1087] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:58:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:56.051604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5927b039-d441-4900-80ff-906c6e468d26 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.081090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.149400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5927b039-d441-4900-80ff-906c6e468d26 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.183218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5f2e49-ebed-4ecf-b682-bb25c236ef3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.183931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f2e49-ebed-4ecf-b682-bb25c236ef3f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.184250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f2e49-ebed-4ecf-b682-bb25c236ef3f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.184563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5f2e49-ebed-4ecf-b682-bb25c236ef3f None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.185350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f2e49-ebed-4ecf-b682-bb25c236ef3f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:56.185659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f2e49-ebed-4ecf-b682-bb25c236ef3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:56.187031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5f2e49-ebed-4ecf-b682-bb25c236ef3f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:56.195177 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 546/1088] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:56.200924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.240137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.289971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b961bb7e-4afd-4772-9ae1-9fd25c083557 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.290182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b961bb7e-4afd-4772-9ae1-9fd25c083557 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.290458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b961bb7e-4afd-4772-9ae1-9fd25c083557 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 547/1089] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 09:58:56.297367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d42f4-2266-4a16-a94a-346f401c2e14 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.297654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d42f4-2266-4a16-a94a-346f401c2e14 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.297931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d42f4-2266-4a16-a94a-346f401c2e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.304273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052dbd23-dc15-4d30-b82d-7e262c0f3576 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.314544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-010d42f4-2266-4a16-a94a-346f401c2e14 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:56.315715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-010d42f4-2266-4a16-a94a-346f401c2e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:56.338881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f669d-843d-46ce-b712-878e773e217c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.339751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f669d-843d-46ce-b712-878e773e217c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f669d-843d-46ce-b712-878e773e217c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.340363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399f669d-843d-46ce-b712-878e773e217c None admin] PATH_INFO: `/v3/users/9a6e4c3d068b48d883f4b20d88846c19` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.341592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399f669d-843d-46ce-b712-878e773e217c None admin] RBAC: Authorizing `identity:get_user(user_id=9a6e4c3d068b48d883f4b20d88846c19)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:56.343036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399f669d-843d-46ce-b712-878e773e217c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:56.344858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 543/1090] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/users/9a6e4c3d068b48d883f4b20d88846c19 => generated 285 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:56.350379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f669d-843d-46ce-b712-878e773e217c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.384243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 548/1091] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:56 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:56.407676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d42f4-2266-4a16-a94a-346f401c2e14 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.431057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.449086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399f669d-843d-46ce-b712-878e773e217c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.452035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.483688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.484340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.484658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.484939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.506668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d42f4-2266-4a16-a94a-346f401c2e14 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.541025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.541765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.542051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.542331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.551974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:56.552920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:58:56.553458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:58:56.561691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:56.562819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:56.563849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 549/1092] 10.222.0.20 () {56 vars in 1120 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:56.568739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.654098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 544/1093] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:58:56.657362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.660465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.663798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.676045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e92510a-edeb-48b7-97b2-854bbd078ebd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.709145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.710039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.710322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.710601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.711348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:56.711657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:56.713068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:56.721424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 550/1094] 10.222.0.20 () {56 vars in 1115 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:56.726936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.758171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd1d78-8174-41f6-b49b-834f5212e059 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.793039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.793834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.794114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.794386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.795543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] RBAC: Authorizing `identity:get_project(project_id=bfdaee025af14785b5f72b76b0d2786b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:56.796995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:56.803858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 545/1095] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:56.804657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.823443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6791cc7-ba86-4685-8e94-3b04f725e105 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.856715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.857511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.857802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.858081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.868460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:56.869906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:56.872995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.874331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:56.888785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:56.889868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:56.890828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:56.890828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:56.890828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:56.890828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:56.890828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 551/1096] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:56.895988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:56.906947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a4d72a-18bb-42e9-9ddf-bb72833fa021 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:56.946061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:56.946720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:56.947008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:56.947290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:56.948115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=bfdaee025af14785b5f72b76b0d2786b, user.id=9a6e4c3d068b48d883f4b20d88846c19) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:56.948451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:56.950502 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:56.956607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 546/1097] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/role_assignments?user.id=9a6e4c3d068b48d883f4b20d88846c19&scope.project.id=bfdaee025af14785b5f72b76b0d2786b&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:56.997284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba92b0b0-07f4-4043-b8aa-641ccdb7ec0d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:57.034641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:57.035319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:57.035606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:57.035887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:57.036660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:57.036972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:57.038387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:57.051955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 552/1098] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:57.057859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:57.159777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64f000a-0bb1-4d28-927b-4f222ea0e26b None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:57.193590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:57.194379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:57.194660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:57.194933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] PATH_INFO: `/v3/users/nova` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:57.197765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:57.204246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:57.205879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 09:58:57.219635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:57.220707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:58:57.222071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 547/1099] 10.222.0.20 () {56 vars in 1093 bytes} [Tue Jul 5 09:58:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:57.226481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:57.281983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:57.325804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250b14b6-d5ab-405b-b41e-02a05349040d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:57.358643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:57.359305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:57.359586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:57.359860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:57.360718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:57.361026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:57.362447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:57.379494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 553/1100] 10.222.0.20 () {56 vars in 1133 bytes} [Tue Jul 5 09:58:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:57.385146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:57.483088 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fdc941-0d43-4945-9e4e-a1deddf1865c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:57.515699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:57.516350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:57.516668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:57.516946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:57.577347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:57.584648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:57.586095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:57.683623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 548/1101] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:58:57 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:58:57.689320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:57.784300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08a3617-1b5f-4fd4-a53f-12f7e0cfbabf None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:57.798920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:57.818666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:57.819442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:57.819721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:57.819992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:57.829686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:57.831060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:58:57.848207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:57.849306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:57.850225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 09:58:57.850225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:58:57.850225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 09:58:57.850225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:57.850225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 554/1102] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 09:58:57.854908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:57.948324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:57.954713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c6c4b2-1b4b-478a-91c3-5a24c2d5d4eb None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:57.988127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:57.988846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:57.989133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:57.989414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:57.990135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:57.990443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:57.991807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:58.002950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 549/1103] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 09:58:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:58:58.008699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:58.107532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc1f50e-324f-4010-872e-4bf2494a5a35 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:58.140255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:58.141070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:58.141360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:58.141640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:58.148442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:58.149886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 09:58:58.163753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:58:58.164694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:58:58.165582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 555/1104] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 09:58:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:58:58.171269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:58.267571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58dc422-ff97-4b07-b030-9d5741e48515 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:58.299839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d5ed29-6a50-45e7-adfb-a5535467a25b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:58.300511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5ed29-6a50-45e7-adfb-a5535467a25b None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:58.300796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5ed29-6a50-45e7-adfb-a5535467a25b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:58.301070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d5ed29-6a50-45e7-adfb-a5535467a25b None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:58.301842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d5ed29-6a50-45e7-adfb-a5535467a25b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:58.302150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d5ed29-6a50-45e7-adfb-a5535467a25b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:58.303559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d5ed29-6a50-45e7-adfb-a5535467a25b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:58.315611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 550/1105] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 09:58:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:58.321515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:58.348413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:58.419667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c81085-f877-49d0-8bc0-3cb17409ac70 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:58.451631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb8c852-de0a-4377-abe0-f8487af0c87d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:58.452286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb8c852-de0a-4377-abe0-f8487af0c87d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:58.452605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb8c852-de0a-4377-abe0-f8487af0c87d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:58.452888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb8c852-de0a-4377-abe0-f8487af0c87d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:58.453712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb8c852-de0a-4377-abe0-f8487af0c87d None admin] RBAC: Adding query filter params (role.id=5d677ad38b8c41579a9b215dadf5562e, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=f99a9fe4b40646d389c3ebf47358956c) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:58:58.454026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb8c852-de0a-4377-abe0-f8487af0c87d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:58.455459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb8c852-de0a-4377-abe0-f8487af0c87d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:58.461248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 556/1106] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:58:58 2022] GET /identity/v3/role_assignments?user.id=f99a9fe4b40646d389c3ebf47358956c&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=5d677ad38b8c41579a9b215dadf5562e => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:58.996087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e4bded-8d06-4d5d-aaff-f81a70bb5e23 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:58.996651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e4bded-8d06-4d5d-aaff-f81a70bb5e23 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:58.996755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e4bded-8d06-4d5d-aaff-f81a70bb5e23 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:58.998107 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 551/1107] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:58:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:58:59.003512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86e761c-c886-44e2-80cc-b1642ec86022 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:59.003963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86e761c-c886-44e2-80cc-b1642ec86022 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:59.004055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86e761c-c886-44e2-80cc-b1642ec86022 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:59.021614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e86e761c-c886-44e2-80cc-b1642ec86022 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:58:59.022794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e86e761c-c886-44e2-80cc-b1642ec86022 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:58:59.089703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 557/1108] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:58:59 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 09:58:59.114218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e4bded-8d06-4d5d-aaff-f81a70bb5e23 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:59.133055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:59.219283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e4bded-8d06-4d5d-aaff-f81a70bb5e23 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:59.254093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf978e5-f868-4eca-9ce7-34818dac17b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:59.254886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf978e5-f868-4eca-9ce7-34818dac17b0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:59.255170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf978e5-f868-4eca-9ce7-34818dac17b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:59.255446 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf978e5-f868-4eca-9ce7-34818dac17b0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:59.256645 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf978e5-f868-4eca-9ce7-34818dac17b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:59.258087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf978e5-f868-4eca-9ce7-34818dac17b0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:59.259705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 552/1109] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:58:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:58:59.265778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86e761c-c886-44e2-80cc-b1642ec86022 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:58:59.287773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:58:59.362566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86e761c-c886-44e2-80cc-b1642ec86022 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:58:59.395608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4573db-6043-420e-96f4-1036218b7573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:58:59.396272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4573db-6043-420e-96f4-1036218b7573 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:58:59.396590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4573db-6043-420e-96f4-1036218b7573 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:58:59.396866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4573db-6043-420e-96f4-1036218b7573 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:58:59.397572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4573db-6043-420e-96f4-1036218b7573 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:58:59.398992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4573db-6043-420e-96f4-1036218b7573 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:58:59.417854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 558/1110] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:58:59 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:59:01.432538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cc2f55-b58a-4bc6-98a9-145c57d9fa9d None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:01.433083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cc2f55-b58a-4bc6-98a9-145c57d9fa9d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:01.433083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cc2f55-b58a-4bc6-98a9-145c57d9fa9d None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:01.434462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 553/1111] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:01.439666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d467d9f6-68ad-475f-a4bd-921247fbefc5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:01.440069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d467d9f6-68ad-475f-a4bd-921247fbefc5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:01.440223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d467d9f6-68ad-475f-a4bd-921247fbefc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:01.456708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d467d9f6-68ad-475f-a4bd-921247fbefc5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:01.457890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d467d9f6-68ad-475f-a4bd-921247fbefc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:01.522002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 559/1112] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:01 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:01.545967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cc2f55-b58a-4bc6-98a9-145c57d9fa9d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:01.627361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:01.643826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cc2f55-b58a-4bc6-98a9-145c57d9fa9d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:01.678749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:01.679595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:01.679876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:01.680151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] PATH_INFO: `/v3/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:01.681637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] RBAC: Authorizing `identity:get_role(role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:01.683055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:01.684399 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 554/1113] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:01 2022] GET /identity/v3/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:01.690356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d467d9f6-68ad-475f-a4bd-921247fbefc5 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:01.698536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:01.787136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d467d9f6-68ad-475f-a4bd-921247fbefc5 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:01.818773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:01.819628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:01.819912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:01.820188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] PATH_INFO: `/v3/users/72df2b7a7e99451b81180b56173e23cc` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:01.833303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] RBAC: Authorizing `identity:get_user(user_id=72df2b7a7e99451b81180b56173e23cc)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:01.834743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:01.836373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 560/1114] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:01 2022] GET /identity/v3/users/72df2b7a7e99451b81180b56173e23cc => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:01.842746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:01.942037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facbdc7d-8dbf-4e56-ab8c-29fa11e70384 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:01.975148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:01.975791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:01.976077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:01.976358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:01.996820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:02.041973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:02.043415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:02.137064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 555/1115] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:59:01 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:59:02.143479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:02.203397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:02.238763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3f5118-1438-495e-885f-ac59e46c45a1 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:02.271768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680a3af-8eb0-41f7-af4e-811a369f2ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:02.272641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680a3af-8eb0-41f7-af4e-811a369f2ded None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:02.272928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680a3af-8eb0-41f7-af4e-811a369f2ded None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:02.273208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680a3af-8eb0-41f7-af4e-811a369f2ded None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:02.274376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680a3af-8eb0-41f7-af4e-811a369f2ded None admin] RBAC: Authorizing `identity:get_project(project_id=bfdaee025af14785b5f72b76b0d2786b)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:02.275804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680a3af-8eb0-41f7-af4e-811a369f2ded None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:02.277515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 561/1116] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:02 2022] GET /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:02.283474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:02.333745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:02.379365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2677b16f-7278-47e2-9a69-d3857649e482 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:02.412326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d9b947-c14d-4552-bea3-ab28299d6cd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:02.412990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d9b947-c14d-4552-bea3-ab28299d6cd3 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:02.413272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d9b947-c14d-4552-bea3-ab28299d6cd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:02.413550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d9b947-c14d-4552-bea3-ab28299d6cd3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:02.414371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d9b947-c14d-4552-bea3-ab28299d6cd3 None admin] RBAC: Adding query filter params (role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=bfdaee025af14785b5f72b76b0d2786b, user.id=72df2b7a7e99451b81180b56173e23cc) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:02.414683 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d9b947-c14d-4552-bea3-ab28299d6cd3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:02.416091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d9b947-c14d-4552-bea3-ab28299d6cd3 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:02.421903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 556/1117] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:59:02 2022] GET /identity/v3/role_assignments?user.id=72df2b7a7e99451b81180b56173e23cc&scope.project.id=bfdaee025af14785b5f72b76b0d2786b&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:04.372916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28e6686-682b-43d2-bcae-b9e0ff42bdc9 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:04.373437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28e6686-682b-43d2-bcae-b9e0ff42bdc9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:04.373527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28e6686-682b-43d2-bcae-b9e0ff42bdc9 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:04.375346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 562/1118] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:04.380550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9e3c1-261b-4420-bdfd-4e8d5a2e4d72 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:04.380965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9e3c1-261b-4420-bdfd-4e8d5a2e4d72 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:04.381112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9e3c1-261b-4420-bdfd-4e8d5a2e4d72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:04.398383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-dfb9e3c1-261b-4420-bdfd-4e8d5a2e4d72 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:04.399567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-dfb9e3c1-261b-4420-bdfd-4e8d5a2e4d72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:04.470444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 557/1119] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:04 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:04.492165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28e6686-682b-43d2-bcae-b9e0ff42bdc9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:04.590526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28e6686-682b-43d2-bcae-b9e0ff42bdc9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:04.623367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:04.628857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:04.629705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:04.629992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:04.630266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] PATH_INFO: `/v3/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:04.631756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] RBAC: Authorizing `identity:get_role(role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:04.633159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:04.634527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 563/1120] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:04 2022] GET /identity/v3/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 223 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:04.640828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb9e3c1-261b-4420-bdfd-4e8d5a2e4d72 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:04.686512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:04.739142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb9e3c1-261b-4420-bdfd-4e8d5a2e4d72 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:04.772207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:04.773165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:04.773453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:04.773732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] PATH_INFO: `/v3/users/72df2b7a7e99451b81180b56173e23cc` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:04.774911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] RBAC: Authorizing `identity:get_user(user_id=72df2b7a7e99451b81180b56173e23cc)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:04.776357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:04.778010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 558/1121] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:04 2022] GET /identity/v3/users/72df2b7a7e99451b81180b56173e23cc => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:04.784407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:04.879376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87447a-0052-4aef-a3c6-26a26e630869 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:04.911891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71652017-ef57-4f64-b211-db36be6e76af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:04.912761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71652017-ef57-4f64-b211-db36be6e76af None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:04.913048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71652017-ef57-4f64-b211-db36be6e76af None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:04.913326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71652017-ef57-4f64-b211-db36be6e76af None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:04.914485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71652017-ef57-4f64-b211-db36be6e76af None admin] RBAC: Authorizing `identity:get_project(project_id=bfdaee025af14785b5f72b76b0d2786b)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:04.915900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71652017-ef57-4f64-b211-db36be6e76af None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:04.917592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 564/1122] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:04 2022] GET /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:04.923921 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:05.020008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c05390-a450-4c36-bc33-7e0def29cab8 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:05.034259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:05.054143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c38cf-7121-4f87-bd7b-185e9652b24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:05.055024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c38cf-7121-4f87-bd7b-185e9652b24d None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:05.055305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c38cf-7121-4f87-bd7b-185e9652b24d None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:05.055580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c38cf-7121-4f87-bd7b-185e9652b24d None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b/users/72df2b7a7e99451b81180b56173e23cc/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:05.057782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c38cf-7121-4f87-bd7b-185e9652b24d None admin] RBAC: Authorizing `identity:create_grant(project_id=bfdaee025af14785b5f72b76b0d2786b, user_id=72df2b7a7e99451b81180b56173e23cc, role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:05.059210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c38cf-7121-4f87-bd7b-185e9652b24d None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:05.072942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 559/1123] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:59:04 2022] PUT /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b/users/72df2b7a7e99451b81180b56173e23cc/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:59:07.036043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ae92b-7ba3-4fc7-bc29-5808126cde5e None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:07.036623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ae92b-7ba3-4fc7-bc29-5808126cde5e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:07.036623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ae92b-7ba3-4fc7-bc29-5808126cde5e None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:07.038030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 565/1124] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:07.043150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab79a03-35df-4b45-806a-cdbc5729385d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:07.043538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab79a03-35df-4b45-806a-cdbc5729385d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:07.043707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab79a03-35df-4b45-806a-cdbc5729385d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:07.060572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5ab79a03-35df-4b45-806a-cdbc5729385d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:07.062345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5ab79a03-35df-4b45-806a-cdbc5729385d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:07.126324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 560/1125] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:07 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:07.149042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912ae92b-7ba3-4fc7-bc29-5808126cde5e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:07.214841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:07.248988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912ae92b-7ba3-4fc7-bc29-5808126cde5e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:07.282588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:07.283435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:07.283722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:07.284004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] PATH_INFO: `/v3/roles/9fc70d9b4ce441be9d89a765290409d2` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:07.285476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] RBAC: Authorizing `identity:get_role(role_id=9fc70d9b4ce441be9d89a765290409d2)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:07.286841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:07.288197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 566/1126] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:07 2022] GET /identity/v3/roles/9fc70d9b4ce441be9d89a765290409d2 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:07.293761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab79a03-35df-4b45-806a-cdbc5729385d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:07.388751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab79a03-35df-4b45-806a-cdbc5729385d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:07.390970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:07.423019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:07.423882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:07.424164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:07.424465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] PATH_INFO: `/v3/users/72df2b7a7e99451b81180b56173e23cc` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:07.425647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] RBAC: Authorizing `identity:get_user(user_id=72df2b7a7e99451b81180b56173e23cc)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:07.427091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:07.428744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 561/1127] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:07 2022] GET /identity/v3/users/72df2b7a7e99451b81180b56173e23cc => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:07.434483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:07.529976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b1fe0e-911b-46ed-a5d7-f8148316527b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:07.563115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:07.563747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:07.564036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:07.564315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:07.567644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:07.629807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:07.631244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:07.726438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 567/1128] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:59:07 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 09:59:07.732264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:07.828835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648bcaf7-8cac-4b10-9b47-095d74b584f9 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:07.860875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca90b4-583f-41ff-980b-d9e14e577f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:07.861721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca90b4-583f-41ff-980b-d9e14e577f26 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:07.862005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca90b4-583f-41ff-980b-d9e14e577f26 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:07.862280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca90b4-583f-41ff-980b-d9e14e577f26 None admin] PATH_INFO: `/v3/projects/bfdaee025af14785b5f72b76b0d2786b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:07.863438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca90b4-583f-41ff-980b-d9e14e577f26 None admin] RBAC: Authorizing `identity:get_project(project_id=bfdaee025af14785b5f72b76b0d2786b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:07.864899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca90b4-583f-41ff-980b-d9e14e577f26 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:07.866553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 562/1129] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:07 2022] GET /identity/v3/projects/bfdaee025af14785b5f72b76b0d2786b => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:07.872326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:07.905801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:07.969120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae81de6-66bf-4597-a1a7-dc1723ab50cd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:08.002235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438ae151-f832-44a0-a4e6-7734815c16c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:08.002883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ae151-f832-44a0-a4e6-7734815c16c0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:08.003171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ae151-f832-44a0-a4e6-7734815c16c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:08.003455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ae151-f832-44a0-a4e6-7734815c16c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:08.004284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ae151-f832-44a0-a4e6-7734815c16c0 None admin] RBAC: Adding query filter params (role.id=9fc70d9b4ce441be9d89a765290409d2, scope.project.id=bfdaee025af14785b5f72b76b0d2786b, user.id=72df2b7a7e99451b81180b56173e23cc) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:08.004637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ae151-f832-44a0-a4e6-7734815c16c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:08.006064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ae151-f832-44a0-a4e6-7734815c16c0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:08.011718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 568/1130] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:59:07 2022] GET /identity/v3/role_assignments?user.id=72df2b7a7e99451b81180b56173e23cc&scope.project.id=bfdaee025af14785b5f72b76b0d2786b&role.id=9fc70d9b4ce441be9d89a765290409d2 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:59:10.007354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0532d917-50cb-480a-b7e8-92db81884f77 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:10.007853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0532d917-50cb-480a-b7e8-92db81884f77 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:10.007952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0532d917-50cb-480a-b7e8-92db81884f77 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:10.009331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 563/1131] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:10.014275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88695402-77ac-4925-96f3-00a21f956a14 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:10.014686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88695402-77ac-4925-96f3-00a21f956a14 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:10.014851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88695402-77ac-4925-96f3-00a21f956a14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:10.031512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-88695402-77ac-4925-96f3-00a21f956a14 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:10.032741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-88695402-77ac-4925-96f3-00a21f956a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:10.096603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 569/1132] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:10 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:10.119674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0532d917-50cb-480a-b7e8-92db81884f77 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:10.178534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:10.236620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0532d917-50cb-480a-b7e8-92db81884f77 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:10.269435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa399dc5-76f5-466a-b79f-95a3cecc72e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:10.270222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa399dc5-76f5-466a-b79f-95a3cecc72e8 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:10.270508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa399dc5-76f5-466a-b79f-95a3cecc72e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:10.270788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa399dc5-76f5-466a-b79f-95a3cecc72e8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:10.272214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa399dc5-76f5-466a-b79f-95a3cecc72e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:10.273683 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa399dc5-76f5-466a-b79f-95a3cecc72e8 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:10.275285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 564/1133] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:59:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:10.280894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88695402-77ac-4925-96f3-00a21f956a14 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:10.309565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:10.375970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88695402-77ac-4925-96f3-00a21f956a14 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:10.408324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2fa8d90-e6f4-4f40-8e4b-26265fcfffa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:10.408988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa8d90-e6f4-4f40-8e4b-26265fcfffa7 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:10.409269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa8d90-e6f4-4f40-8e4b-26265fcfffa7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:10.409544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2fa8d90-e6f4-4f40-8e4b-26265fcfffa7 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:10.410233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fa8d90-e6f4-4f40-8e4b-26265fcfffa7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:10.411645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2fa8d90-e6f4-4f40-8e4b-26265fcfffa7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:10.436461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 570/1134] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:59:10 2022] POST /identity/v3/projects => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:59:12.472243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250b711b-db69-4380-b23c-597eb8811750 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:12.472825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250b711b-db69-4380-b23c-597eb8811750 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:12.472825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250b711b-db69-4380-b23c-597eb8811750 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:12.474205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 565/1135] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:12.479394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b90115-8b85-4150-bf01-9d869d1c0ca9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:12.479808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b90115-8b85-4150-bf01-9d869d1c0ca9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:12.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b90115-8b85-4150-bf01-9d869d1c0ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:12.496834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b6b90115-8b85-4150-bf01-9d869d1c0ca9 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:12.498007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b6b90115-8b85-4150-bf01-9d869d1c0ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:12.562842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 571/1136] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:12.587424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250b711b-db69-4380-b23c-597eb8811750 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:12.686561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250b711b-db69-4380-b23c-597eb8811750 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:12.696006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:12.720698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7864fc-a664-48d4-8a60-8ea386d1175a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:12.721475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7864fc-a664-48d4-8a60-8ea386d1175a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:12.721751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7864fc-a664-48d4-8a60-8ea386d1175a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:12.722019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7864fc-a664-48d4-8a60-8ea386d1175a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:12.723213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7864fc-a664-48d4-8a60-8ea386d1175a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:12.724728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7864fc-a664-48d4-8a60-8ea386d1175a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:12.726365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 566/1137] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 09:59:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:12.732120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b90115-8b85-4150-bf01-9d869d1c0ca9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:12.745843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:12.828619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b90115-8b85-4150-bf01-9d869d1c0ca9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:12.860027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3782b1ee-3021-4238-8946-293a7adcba06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:12.860705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782b1ee-3021-4238-8946-293a7adcba06 None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:12.860987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782b1ee-3021-4238-8946-293a7adcba06 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:12.861262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782b1ee-3021-4238-8946-293a7adcba06 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:12.861962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3782b1ee-3021-4238-8946-293a7adcba06 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:12.863382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3782b1ee-3021-4238-8946-293a7adcba06 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:12.882226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 572/1138] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:59:12 2022] POST /identity/v3/users => generated 286 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 09:59:14.887366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520da839-5fca-446c-a591-f5a9b590c268 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:14.887889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520da839-5fca-446c-a591-f5a9b590c268 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:14.887889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520da839-5fca-446c-a591-f5a9b590c268 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:14.889300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 567/1139] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:14.894375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9001b103-6c0a-47b4-b3da-a3ded74dd2d7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:14.894753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9001b103-6c0a-47b4-b3da-a3ded74dd2d7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:14.894931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9001b103-6c0a-47b4-b3da-a3ded74dd2d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:14.911861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9001b103-6c0a-47b4-b3da-a3ded74dd2d7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:14.913061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9001b103-6c0a-47b4-b3da-a3ded74dd2d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:14.972506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:14.978490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 573/1140] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:14 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:15.001451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520da839-5fca-446c-a591-f5a9b590c268 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:15.101696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520da839-5fca-446c-a591-f5a9b590c268 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:15.134204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:15.134975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:15.135259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:15.135538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:15.145436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:59:15.153157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:59:15.154135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:59:15.155096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 568/1141] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:59:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:59:15.160661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9001b103-6c0a-47b4-b3da-a3ded74dd2d7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:15.255494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9001b103-6c0a-47b4-b3da-a3ded74dd2d7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:15.288809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:15.289466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:15.289755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:15.290038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:15.290782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:15.291093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:15.292487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:15.300288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 574/1142] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:59:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:15.305756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:15.339914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:15.401721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c467f96-3936-4e87-910d-0bd85f7c34da None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:15.433674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:15.434512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:15.434794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:15.435068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] PATH_INFO: `/v3/users/ee186fc44a1048d2b41f75981c87e4b4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:15.447585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] RBAC: Authorizing `identity:get_user(user_id=ee186fc44a1048d2b41f75981c87e4b4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:15.449054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:15.450653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 569/1143] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:15 2022] GET /identity/v3/users/ee186fc44a1048d2b41f75981c87e4b4 => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:15.456757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:15.513647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:15.553687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf42d7-2a8e-4ed7-af5a-f812f5d42a5b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:15.586067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:15.586701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:15.586985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:15.587261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:15.651208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:15.652719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:15.707691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:15.747408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 575/1144] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:59:15 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:59:15.755296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:15.845666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:15.852730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04d96bd-594f-4d02-96f9-77284f7dae12 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:15.885513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027d477f-360a-4f7d-8fbf-3a8b827a2e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:15.886363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d477f-360a-4f7d-8fbf-3a8b827a2e9a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:15.886650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d477f-360a-4f7d-8fbf-3a8b827a2e9a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:15.886929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027d477f-360a-4f7d-8fbf-3a8b827a2e9a None admin] PATH_INFO: `/v3/projects/6daceadd263a4786ba80dc29df5c21f8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:15.888100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027d477f-360a-4f7d-8fbf-3a8b827a2e9a None admin] RBAC: Authorizing `identity:get_project(project_id=6daceadd263a4786ba80dc29df5c21f8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:15.889572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027d477f-360a-4f7d-8fbf-3a8b827a2e9a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:15.891827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 570/1145] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:15 2022] GET /identity/v3/projects/6daceadd263a4786ba80dc29df5c21f8 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:59:15.898099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:15.995702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7d7b16-cb79-4ce1-8cf4-9bc097d52003 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:16.028708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1567c534-0932-48b8-ab23-cd6f40ebc8f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:16.029357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1567c534-0932-48b8-ab23-cd6f40ebc8f7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:16.029646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1567c534-0932-48b8-ab23-cd6f40ebc8f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:16.029923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1567c534-0932-48b8-ab23-cd6f40ebc8f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:16.030906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1567c534-0932-48b8-ab23-cd6f40ebc8f7 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=6daceadd263a4786ba80dc29df5c21f8, user.id=ee186fc44a1048d2b41f75981c87e4b4) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:16.031225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1567c534-0932-48b8-ab23-cd6f40ebc8f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:16.032679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1567c534-0932-48b8-ab23-cd6f40ebc8f7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:16.037930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 576/1146] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:59:15 2022] GET /identity/v3/role_assignments?user.id=ee186fc44a1048d2b41f75981c87e4b4&scope.project.id=6daceadd263a4786ba80dc29df5c21f8&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:59:18.011042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9187635-05f5-4226-810d-d9a814a20fb5 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:18.011510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9187635-05f5-4226-810d-d9a814a20fb5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:18.011707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9187635-05f5-4226-810d-d9a814a20fb5 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:18.013002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 571/1147] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:18.018719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5643d-d51c-49d0-8d86-4c9605141f8a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:18.019132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5643d-d51c-49d0-8d86-4c9605141f8a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:18.019277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5643d-d51c-49d0-8d86-4c9605141f8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:18.035899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-dcc5643d-d51c-49d0-8d86-4c9605141f8a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:18.037116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dcc5643d-d51c-49d0-8d86-4c9605141f8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:18.053703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:18.101624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 577/1148] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:18 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:18.123549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9187635-05f5-4226-810d-d9a814a20fb5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:18.223008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9187635-05f5-4226-810d-d9a814a20fb5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:18.256321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:18.257125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:18.257412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:18.257691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:18.267557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:59:18.275504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:59:18.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:59:18.277819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 572/1149] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:59:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:59:18.282561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5643d-d51c-49d0-8d86-4c9605141f8a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:18.377079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5643d-d51c-49d0-8d86-4c9605141f8a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:18.409016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:18.409664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:18.409951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:18.410229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:18.410966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:18.411275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:18.412671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:18.420415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 578/1150] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:59:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:59:18.426280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:18.451039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:18.525145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27306146-41fc-4c07-8e82-2ee559069023 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:18.557668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:18.558516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:18.558800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:18.559085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] PATH_INFO: `/v3/users/ee186fc44a1048d2b41f75981c87e4b4` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:18.560252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] RBAC: Authorizing `identity:get_user(user_id=ee186fc44a1048d2b41f75981c87e4b4)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:18.561724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:18.563887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 573/1151] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:18 2022] GET /identity/v3/users/ee186fc44a1048d2b41f75981c87e4b4 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:18.569571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:18.587193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:18.665176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0757fe03-4d46-4883-88c3-5d5febc11b1f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:18.701257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80e3f29-67f6-4e1e-9886-e58aa88dab7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:18.702111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80e3f29-67f6-4e1e-9886-e58aa88dab7f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:18.702398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80e3f29-67f6-4e1e-9886-e58aa88dab7f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:18.702704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80e3f29-67f6-4e1e-9886-e58aa88dab7f None admin] PATH_INFO: `/v3/projects/6daceadd263a4786ba80dc29df5c21f8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:18.703861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80e3f29-67f6-4e1e-9886-e58aa88dab7f None admin] RBAC: Authorizing `identity:get_project(project_id=6daceadd263a4786ba80dc29df5c21f8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:18.705314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80e3f29-67f6-4e1e-9886-e58aa88dab7f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:18.707004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 579/1152] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:18 2022] GET /identity/v3/projects/6daceadd263a4786ba80dc29df5c21f8 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:59:18.712862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:18.802257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:18.812056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a225fd2d-db51-491b-bc72-792465efecc5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:18.844567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d181e5a-2e27-42e3-bd82-4a95a60dfbfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:18.845445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d181e5a-2e27-42e3-bd82-4a95a60dfbfb None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:18.845733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d181e5a-2e27-42e3-bd82-4a95a60dfbfb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:18.846012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d181e5a-2e27-42e3-bd82-4a95a60dfbfb None admin] PATH_INFO: `/v3/projects/6daceadd263a4786ba80dc29df5c21f8/users/ee186fc44a1048d2b41f75981c87e4b4/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:18.847960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d181e5a-2e27-42e3-bd82-4a95a60dfbfb None admin] RBAC: Authorizing `identity:create_grant(project_id=6daceadd263a4786ba80dc29df5c21f8, user_id=ee186fc44a1048d2b41f75981c87e4b4, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:18.849424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d181e5a-2e27-42e3-bd82-4a95a60dfbfb None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:18.862737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 574/1153] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:59:18 2022] PUT /identity/v3/projects/6daceadd263a4786ba80dc29df5c21f8/users/ee186fc44a1048d2b41f75981c87e4b4/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:59:20.846578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b930ac26-8059-4c16-ae3e-d8c182a991bd None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:20.847126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b930ac26-8059-4c16-ae3e-d8c182a991bd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:20.847126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b930ac26-8059-4c16-ae3e-d8c182a991bd None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:20.848528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 580/1154] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:20.853643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12055338-68c9-49d5-9f59-0b6f96d25539 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:20.854039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12055338-68c9-49d5-9f59-0b6f96d25539 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:20.854208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12055338-68c9-49d5-9f59-0b6f96d25539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:20.870974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-12055338-68c9-49d5-9f59-0b6f96d25539 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:20.872151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-12055338-68c9-49d5-9f59-0b6f96d25539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:20.936889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 575/1155] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:20 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:20.962382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b930ac26-8059-4c16-ae3e-d8c182a991bd None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:21.044666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:21.061524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b930ac26-8059-4c16-ae3e-d8c182a991bd None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:21.093804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:21.094580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:21.094860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:21.095133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:21.104917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:59:21.112825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:59:21.113900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:59:21.119459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 581/1156] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:59:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:59:21.120318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12055338-68c9-49d5-9f59-0b6f96d25539 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:21.196960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:21.216552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12055338-68c9-49d5-9f59-0b6f96d25539 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:21.249246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:21.249926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:21.250210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:21.250486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:21.251229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:21.251537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:21.252944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:21.260747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 576/1157] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:59:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:21.266971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:21.362860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e42ccc-7656-4ef5-b7d4-9fd4a3218d0e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:21.394403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:21.395246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:21.395531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:21.395807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] PATH_INFO: `/v3/users/ee186fc44a1048d2b41f75981c87e4b4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:21.397246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] RBAC: Authorizing `identity:get_user(user_id=ee186fc44a1048d2b41f75981c87e4b4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:21.398661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:21.400250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 582/1158] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:21 2022] GET /identity/v3/users/ee186fc44a1048d2b41f75981c87e4b4 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:21.405694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:21.501716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ed97dd-d0fd-4e88-8e25-c5c14e802be7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:21.535048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:21.535685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:21.535975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:21.536253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:21.547708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:21.601845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:21.603283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:21.698832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 577/1159] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:59:21 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:59:21.705158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:21.719842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:21.801024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad44ceab-9331-4317-b6c0-ed4c034b556f None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:21.833589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118df664-047d-4824-bf97-a08a3f417ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:21.834435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118df664-047d-4824-bf97-a08a3f417ff2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:21.834722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118df664-047d-4824-bf97-a08a3f417ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:21.834999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118df664-047d-4824-bf97-a08a3f417ff2 None admin] PATH_INFO: `/v3/projects/6daceadd263a4786ba80dc29df5c21f8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:21.836155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118df664-047d-4824-bf97-a08a3f417ff2 None admin] RBAC: Authorizing `identity:get_project(project_id=6daceadd263a4786ba80dc29df5c21f8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:21.837610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118df664-047d-4824-bf97-a08a3f417ff2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:21.839284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 583/1160] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:21 2022] GET /identity/v3/projects/6daceadd263a4786ba80dc29df5c21f8 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:21.845064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:21.886339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:21.943084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4575076-b948-46c7-b9ac-f1536c3a9524 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:21.975324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912185e5-77b1-4a5a-8745-f1649ebfa8cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:21.975909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912185e5-77b1-4a5a-8745-f1649ebfa8cc None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:21.976193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912185e5-77b1-4a5a-8745-f1649ebfa8cc None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:21.976497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912185e5-77b1-4a5a-8745-f1649ebfa8cc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:21.977324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912185e5-77b1-4a5a-8745-f1649ebfa8cc None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=6daceadd263a4786ba80dc29df5c21f8, user.id=ee186fc44a1048d2b41f75981c87e4b4) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:21.977637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912185e5-77b1-4a5a-8745-f1649ebfa8cc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:21.979061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912185e5-77b1-4a5a-8745-f1649ebfa8cc None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:21.985247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 578/1161] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:59:21 2022] GET /identity/v3/role_assignments?user.id=ee186fc44a1048d2b41f75981c87e4b4&scope.project.id=6daceadd263a4786ba80dc29df5c21f8&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:23.982366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b53358-e617-4284-82f3-cdc38e446d44 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:23.982830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b53358-e617-4284-82f3-cdc38e446d44 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:23.982945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b53358-e617-4284-82f3-cdc38e446d44 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:23.984947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 584/1162] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:23.990119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1b5a9-7bea-4cfc-8563-be54778b8e08 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:23.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1b5a9-7bea-4cfc-8563-be54778b8e08 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:23.990676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b1b5a9-7bea-4cfc-8563-be54778b8e08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:24.007571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b9b1b5a9-7bea-4cfc-8563-be54778b8e08 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:24.008786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b9b1b5a9-7bea-4cfc-8563-be54778b8e08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:24.073110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 579/1163] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:24.095532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b53358-e617-4284-82f3-cdc38e446d44 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:24.175917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:24.195864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b53358-e617-4284-82f3-cdc38e446d44 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:24.228945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:24.229585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:24.229879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:24.230160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:24.294674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:24.296119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:24.371242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:24.391353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 585/1164] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:59:24 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:59:24.397469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b1b5a9-7bea-4cfc-8563-be54778b8e08 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:24.430774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:24.493426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b1b5a9-7bea-4cfc-8563-be54778b8e08 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:24.526396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25db5dc4-9abb-4a58-9042-cf5308c763b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:24.527183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db5dc4-9abb-4a58-9042-cf5308c763b0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:24.527471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db5dc4-9abb-4a58-9042-cf5308c763b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:24.527751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25db5dc4-9abb-4a58-9042-cf5308c763b0 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:24.535138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25db5dc4-9abb-4a58-9042-cf5308c763b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:24.536599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25db5dc4-9abb-4a58-9042-cf5308c763b0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-25db5dc4-9abb-4a58-9042-cf5308c763b0 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:59:24.550579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:59:24.551818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:59:24.552893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 580/1165] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 09:59:24 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:59:24.557622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:24.652321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b6ee55-8a37-40eb-8bca-00d4e3f811cf None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:24.684198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcc51a1-a254-4515-8899-112deec293db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:24.684851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcc51a1-a254-4515-8899-112deec293db None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:24.685136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcc51a1-a254-4515-8899-112deec293db None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:24.685410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcc51a1-a254-4515-8899-112deec293db None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:24.686111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcc51a1-a254-4515-8899-112deec293db None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:24.686415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcc51a1-a254-4515-8899-112deec293db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:24.688295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcc51a1-a254-4515-8899-112deec293db None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:24.695167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 586/1166] 10.222.0.20 () {56 vars in 1115 bytes} [Tue Jul 5 09:59:24 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:59:26.660271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620818c6-46bf-42fb-8936-afa1058182ec None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:26.660833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620818c6-46bf-42fb-8936-afa1058182ec None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:26.660938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620818c6-46bf-42fb-8936-afa1058182ec None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:26.662236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 581/1167] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:26.667478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d5ae6-8d66-42d0-a725-a90cfb8e9010 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:26.667905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d5ae6-8d66-42d0-a725-a90cfb8e9010 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:26.668043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27d5ae6-8d66-42d0-a725-a90cfb8e9010 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:26.685580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c27d5ae6-8d66-42d0-a725-a90cfb8e9010 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:26.686767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c27d5ae6-8d66-42d0-a725-a90cfb8e9010 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:26.725291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:26.752012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 587/1168] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:26 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:26.775666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620818c6-46bf-42fb-8936-afa1058182ec None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:26.868816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:26.875400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620818c6-46bf-42fb-8936-afa1058182ec None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:26.908192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47d4453-ff5b-4b78-90f5-ac6523e3a7ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:26.908875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47d4453-ff5b-4b78-90f5-ac6523e3a7ec None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:26.909161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47d4453-ff5b-4b78-90f5-ac6523e3a7ec None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:26.909439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47d4453-ff5b-4b78-90f5-ac6523e3a7ec None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:26.910165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47d4453-ff5b-4b78-90f5-ac6523e3a7ec None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:26.911574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47d4453-ff5b-4b78-90f5-ac6523e3a7ec None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:26.925759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 582/1169] 10.222.0.20 () {60 vars in 1138 bytes} [Tue Jul 5 09:59:26 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:59:28.932112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4cb7f-ba9e-4fae-8604-e5651d6304bf None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:28.932567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4cb7f-ba9e-4fae-8604-e5651d6304bf None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:28.932704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd4cb7f-ba9e-4fae-8604-e5651d6304bf None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:28.934123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 588/1170] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:28.940644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4f2d2-8047-4185-b18c-63d1851136cf None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:28.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4f2d2-8047-4185-b18c-63d1851136cf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:28.941210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c4f2d2-8047-4185-b18c-63d1851136cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:28.957702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-02c4f2d2-8047-4185-b18c-63d1851136cf None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:28.958889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-02c4f2d2-8047-4185-b18c-63d1851136cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:29.024342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 583/1171] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:28 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:29.049067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd4cb7f-ba9e-4fae-8604-e5651d6304bf None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:29.146132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd4cb7f-ba9e-4fae-8604-e5651d6304bf None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:29.178819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e8bb42-8e98-4362-88c0-270ea5aad93e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:29.179657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8bb42-8e98-4362-88c0-270ea5aad93e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:29.179968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8bb42-8e98-4362-88c0-270ea5aad93e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:29.180242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e8bb42-8e98-4362-88c0-270ea5aad93e None admin] PATH_INFO: `/v3/domains/afaddd23ae5c401d924d31a58259b7a8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:29.190181 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e8bb42-8e98-4362-88c0-270ea5aad93e None admin] RBAC: Authorizing `identity:get_domain(domain_id=afaddd23ae5c401d924d31a58259b7a8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:29.191622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e8bb42-8e98-4362-88c0-270ea5aad93e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:29.193242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 589/1172] 10.222.0.20 () {56 vars in 1183 bytes} [Tue Jul 5 09:59:29 2022] GET /identity/v3/domains/afaddd23ae5c401d924d31a58259b7a8 => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 09:59:29.198914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4f2d2-8047-4185-b18c-63d1851136cf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:29.280164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:29.297355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c4f2d2-8047-4185-b18c-63d1851136cf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:29.330737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ba483f-4229-4cb9-b4e9-88a7a8b2c61c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:29.331386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ba483f-4229-4cb9-b4e9-88a7a8b2c61c None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:29.331671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ba483f-4229-4cb9-b4e9-88a7a8b2c61c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:29.331948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ba483f-4229-4cb9-b4e9-88a7a8b2c61c None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:29.332673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ba483f-4229-4cb9-b4e9-88a7a8b2c61c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:29.334103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ba483f-4229-4cb9-b4e9-88a7a8b2c61c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:29.360072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 584/1173] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 09:59:29 2022] POST /identity/v3/projects => generated 361 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:59:31.377211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617dfd3-d0d3-4643-bd3f-11eeb26c8743 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:31.377764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617dfd3-d0d3-4643-bd3f-11eeb26c8743 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:31.377764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617dfd3-d0d3-4643-bd3f-11eeb26c8743 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:31.379160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 590/1174] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:31.384164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691a92ce-0060-48c9-b47f-082f5566a926 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:31.384561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691a92ce-0060-48c9-b47f-082f5566a926 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:31.384753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691a92ce-0060-48c9-b47f-082f5566a926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:31.401717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-691a92ce-0060-48c9-b47f-082f5566a926 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:31.402889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-691a92ce-0060-48c9-b47f-082f5566a926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:31.467255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 585/1175] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:31 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:31.490974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0617dfd3-d0d3-4643-bd3f-11eeb26c8743 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:31.526384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:31.590781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0617dfd3-d0d3-4643-bd3f-11eeb26c8743 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:31.622766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210b5ca3-61a7-4212-be9a-e11aa69f54ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:31.623603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210b5ca3-61a7-4212-be9a-e11aa69f54ba None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:31.623888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210b5ca3-61a7-4212-be9a-e11aa69f54ba None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:31.624166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210b5ca3-61a7-4212-be9a-e11aa69f54ba None admin] PATH_INFO: `/v3/domains/afaddd23ae5c401d924d31a58259b7a8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:31.625859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210b5ca3-61a7-4212-be9a-e11aa69f54ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=afaddd23ae5c401d924d31a58259b7a8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:31.627334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210b5ca3-61a7-4212-be9a-e11aa69f54ba None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:31.628908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 591/1176] 10.222.0.20 () {56 vars in 1183 bytes} [Tue Jul 5 09:59:31 2022] GET /identity/v3/domains/afaddd23ae5c401d924d31a58259b7a8 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:31.634406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691a92ce-0060-48c9-b47f-082f5566a926 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:31.698401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:31.729993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691a92ce-0060-48c9-b47f-082f5566a926 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:31.762762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf7b35-4330-4019-84bd-1be1926a7b9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:31.763411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf7b35-4330-4019-84bd-1be1926a7b9b None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:31.763692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf7b35-4330-4019-84bd-1be1926a7b9b None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:31.763966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf7b35-4330-4019-84bd-1be1926a7b9b None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:31.764689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf7b35-4330-4019-84bd-1be1926a7b9b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:31.766136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf7b35-4330-4019-84bd-1be1926a7b9b None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:31.784957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 586/1177] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 09:59:31 2022] POST /identity/v3/users => generated 311 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 09:59:33.836276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6964e050-3cb5-4afc-8708-78c38b17d822 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:33.836928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6964e050-3cb5-4afc-8708-78c38b17d822 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:33.836928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6964e050-3cb5-4afc-8708-78c38b17d822 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:33.838263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 592/1178] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:33.843295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400ad95-92f9-4e4a-a531-8b405b8ce333 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:33.844234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400ad95-92f9-4e4a-a531-8b405b8ce333 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:33.844365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4400ad95-92f9-4e4a-a531-8b405b8ce333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:33.861193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4400ad95-92f9-4e4a-a531-8b405b8ce333 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:33.862389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4400ad95-92f9-4e4a-a531-8b405b8ce333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:33.926129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 587/1179] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:33 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:33.948830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6964e050-3cb5-4afc-8708-78c38b17d822 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:34.046265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6964e050-3cb5-4afc-8708-78c38b17d822 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:34.047761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:34.079723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:34.080524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:34.080815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:34.081093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:34.090823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:59:34.098358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:59:34.099523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:59:34.100870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 593/1180] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:59:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:59:34.105314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400ad95-92f9-4e4a-a531-8b405b8ce333 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:34.159005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:34.202510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4400ad95-92f9-4e4a-a531-8b405b8ce333 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:34.234741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:34.235392 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:34.235672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:34.235942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:34.236723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:34.237027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:34.238385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:34.246094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 588/1181] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:59:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:34.252148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:34.348055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e5657e-9872-4e42-8cd1-9722d5305b7d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:34.380010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:34.380887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:34.381175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:34.381452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] PATH_INFO: `/v3/users/45a04824818a47d29efb86daf4b1faeb` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:34.393917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] RBAC: Authorizing `identity:get_user(user_id=45a04824818a47d29efb86daf4b1faeb)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:34.395362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:34.397007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 594/1182] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:34 2022] GET /identity/v3/users/45a04824818a47d29efb86daf4b1faeb => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:34.402601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:34.499268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08b8e44-0a3e-472a-9607-8d9d26c43939 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:34.514571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:34.532675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:34.533310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:34.533595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:34.533872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:34.598825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:34.600192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:34.694925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 589/1183] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:59:34 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:59:34.700711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:34.722532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:34.798189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b372c2f8-6715-4ec5-a063-cbdd51453267 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:34.836111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ac34d-97ab-4506-84b1-aca009afa87b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:34.836992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ac34d-97ab-4506-84b1-aca009afa87b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:34.837527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ac34d-97ab-4506-84b1-aca009afa87b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:34.837527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ac34d-97ab-4506-84b1-aca009afa87b None admin] PATH_INFO: `/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:34.838719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ac34d-97ab-4506-84b1-aca009afa87b None admin] RBAC: Authorizing `identity:get_project(project_id=17cdb557ddfd4a3e9d7a2684164425c6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:34.840151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ac34d-97ab-4506-84b1-aca009afa87b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:34.841877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 595/1184] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:34 2022] GET /identity/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6 => generated 361 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:34.848663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:34.865508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:34.944830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4c3d90-daad-4ddf-87d5-db987d587c97 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:34.978119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113e16ed-bafa-4e28-b9c6-8e869c878dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:34.978763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e16ed-bafa-4e28-b9c6-8e869c878dfa None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:34.979042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e16ed-bafa-4e28-b9c6-8e869c878dfa None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:34.979316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113e16ed-bafa-4e28-b9c6-8e869c878dfa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:34.980135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113e16ed-bafa-4e28-b9c6-8e869c878dfa None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=17cdb557ddfd4a3e9d7a2684164425c6, user.id=45a04824818a47d29efb86daf4b1faeb) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:34.980620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113e16ed-bafa-4e28-b9c6-8e869c878dfa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:34.982056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113e16ed-bafa-4e28-b9c6-8e869c878dfa None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:34.987320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 590/1185] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:59:34 2022] GET /identity/v3/role_assignments?user.id=45a04824818a47d29efb86daf4b1faeb&scope.project.id=17cdb557ddfd4a3e9d7a2684164425c6&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:36.948377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d6a872-46fb-4f6a-a3f5-455bcf7776f2 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:36.948873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d6a872-46fb-4f6a-a3f5-455bcf7776f2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:36.948972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d6a872-46fb-4f6a-a3f5-455bcf7776f2 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:36.950357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 596/1186] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:36.956159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a30bef9-6aeb-439d-aeba-9057a776b0f2 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:36.956610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a30bef9-6aeb-439d-aeba-9057a776b0f2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:36.956746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a30bef9-6aeb-439d-aeba-9057a776b0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:36.973553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2a30bef9-6aeb-439d-aeba-9057a776b0f2 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:36.974717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2a30bef9-6aeb-439d-aeba-9057a776b0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:37.039321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 591/1187] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:37.060913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d6a872-46fb-4f6a-a3f5-455bcf7776f2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:37.146111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:37.159640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d6a872-46fb-4f6a-a3f5-455bcf7776f2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:37.191341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:37.192118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:37.192428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:37.192711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:37.202430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:59:37.210574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:59:37.211647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 09:59:37.212551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 597/1188] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:59:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:59:37.217716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a30bef9-6aeb-439d-aeba-9057a776b0f2 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:37.229555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:37.314286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a30bef9-6aeb-439d-aeba-9057a776b0f2 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:37.347371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:37.348025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:37.348313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:37.348624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:37.349367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:37.349675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:37.351039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:37.358909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 592/1189] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:59:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:37.364482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:37.460628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7846011d-7f9a-4fde-b130-8fd6aa0cad18 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:37.491775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:37.492648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:37.492934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:37.493214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] PATH_INFO: `/v3/users/45a04824818a47d29efb86daf4b1faeb` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:37.494389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] RBAC: Authorizing `identity:get_user(user_id=45a04824818a47d29efb86daf4b1faeb)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:37.495813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:37.497453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 598/1190] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:37 2022] GET /identity/v3/users/45a04824818a47d29efb86daf4b1faeb => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:37.503440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:37.571807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:37.601244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa01a2f-a1a0-4d72-ab30-982fc91acff6 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:37.633387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cfdbb4-9cca-407a-8963-d141fe9ebdda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:37.634231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cfdbb4-9cca-407a-8963-d141fe9ebdda None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:37.634516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cfdbb4-9cca-407a-8963-d141fe9ebdda None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:37.634793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cfdbb4-9cca-407a-8963-d141fe9ebdda None admin] PATH_INFO: `/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:37.635958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cfdbb4-9cca-407a-8963-d141fe9ebdda None admin] RBAC: Authorizing `identity:get_project(project_id=17cdb557ddfd4a3e9d7a2684164425c6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:37.637406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cfdbb4-9cca-407a-8963-d141fe9ebdda None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:37.639084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 593/1191] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:37 2022] GET /identity/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:37.645485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:37.663963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:37.743992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780b98b9-0a84-4839-a595-0125ec888221 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:37.776621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69663d5b-4a4d-4137-b743-13bcce4ef3b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:37.777494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69663d5b-4a4d-4137-b743-13bcce4ef3b0 None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:37.777777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69663d5b-4a4d-4137-b743-13bcce4ef3b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:37.778055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69663d5b-4a4d-4137-b743-13bcce4ef3b0 None admin] PATH_INFO: `/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6/users/45a04824818a47d29efb86daf4b1faeb/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:37.779981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69663d5b-4a4d-4137-b743-13bcce4ef3b0 None admin] RBAC: Authorizing `identity:create_grant(project_id=17cdb557ddfd4a3e9d7a2684164425c6, user_id=45a04824818a47d29efb86daf4b1faeb, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:37.781466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69663d5b-4a4d-4137-b743-13bcce4ef3b0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:37.793286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 599/1192] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 09:59:37 2022] PUT /identity/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6/users/45a04824818a47d29efb86daf4b1faeb/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 09:59:39.770280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b12c22-a450-49b2-bc0f-6468aa0dee34 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:39.770801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b12c22-a450-49b2-bc0f-6468aa0dee34 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:39.770912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b12c22-a450-49b2-bc0f-6468aa0dee34 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:39.772178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 594/1193] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 09:59:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 09:59:39.777522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf83dbd-750a-451f-b3bf-50358aa402b2 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:39.777955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf83dbd-750a-451f-b3bf-50358aa402b2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:39.778093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf83dbd-750a-451f-b3bf-50358aa402b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:39.795030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ebf83dbd-750a-451f-b3bf-50358aa402b2 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 09:59:39.796221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ebf83dbd-750a-451f-b3bf-50358aa402b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 09:59:39.861741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 600/1194] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 09:59:39 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 09:59:39.884507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b12c22-a450-49b2-bc0f-6468aa0dee34 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:39.984166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b12c22-a450-49b2-bc0f-6468aa0dee34 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:40.016808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:40.017589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:40.017875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:40.018154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:40.025869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:40.029017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 09:59:40.037447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 09:59:40.038544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 09:59:40.039575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 595/1195] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 09:59:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 09:59:40.044636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf83dbd-750a-451f-b3bf-50358aa402b2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:40.052665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:40.141771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf83dbd-750a-451f-b3bf-50358aa402b2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:40.175164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:40.175822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:40.176109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:40.176408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:40.177161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:40.177473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:40.178844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:40.186674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 601/1196] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 09:59:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:40.192344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:40.290247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70db3b8-7843-4cb0-80b4-942ff6c7c762 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:40.322672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:40.323510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:40.323795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:40.324071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] PATH_INFO: `/v3/users/45a04824818a47d29efb86daf4b1faeb` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:40.325271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] RBAC: Authorizing `identity:get_user(user_id=45a04824818a47d29efb86daf4b1faeb)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:40.326685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:40.328823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 596/1197] 10.222.0.20 () {56 vars in 1177 bytes} [Tue Jul 5 09:59:40 2022] GET /identity/v3/users/45a04824818a47d29efb86daf4b1faeb => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:40.334356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:40.398070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:40.429442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563f77d9-badc-4853-900f-fc8497f73a57 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:40.461626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:40.462255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:40.462539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:40.462814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:40.526587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:40.528002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:40.588861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:40.620921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 602/1198] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 09:59:40 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 09:59:40.627123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:40.693792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:40.725903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be21e75f-1be0-4462-a011-49c5c9e53fba None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:40.759530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0da4b2-3f63-46b1-8215-d662d02622b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:40.760378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0da4b2-3f63-46b1-8215-d662d02622b4 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:40.760687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0da4b2-3f63-46b1-8215-d662d02622b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:40.760965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0da4b2-3f63-46b1-8215-d662d02622b4 None admin] PATH_INFO: `/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:40.762122 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0da4b2-3f63-46b1-8215-d662d02622b4 None admin] RBAC: Authorizing `identity:get_project(project_id=17cdb557ddfd4a3e9d7a2684164425c6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:40.763559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0da4b2-3f63-46b1-8215-d662d02622b4 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:40.765274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 597/1199] 10.222.0.20 () {56 vars in 1186 bytes} [Tue Jul 5 09:59:40 2022] GET /identity/v3/projects/17cdb557ddfd4a3e9d7a2684164425c6 => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:40.771292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 09:59:40.875786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eee5f31-5308-42d6-b4fd-e513afbba9e9 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 09:59:40.907756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f31c5c-a1cf-4853-b229-cf0904e0dffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 09:59:40.908420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f31c5c-a1cf-4853-b229-cf0904e0dffd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 09:59:40.908703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f31c5c-a1cf-4853-b229-cf0904e0dffd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 09:59:40.908977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f31c5c-a1cf-4853-b229-cf0904e0dffd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 09:59:40.909800 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f31c5c-a1cf-4853-b229-cf0904e0dffd None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=17cdb557ddfd4a3e9d7a2684164425c6, user.id=45a04824818a47d29efb86daf4b1faeb) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 09:59:40.910112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f31c5c-a1cf-4853-b229-cf0904e0dffd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 09:59:40.911527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f31c5c-a1cf-4853-b229-cf0904e0dffd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 09:59:40.917153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 603/1200] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 09:59:40 2022] GET /identity/v3/role_assignments?user.id=45a04824818a47d29efb86daf4b1faeb&scope.project.id=17cdb557ddfd4a3e9d7a2684164425c6&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 09:59:50.755485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=32 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 09:59:50.914551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=84 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:35.352950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d073e7f-5d9a-4b80-8170-1ec18c224d7b None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:35.353539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d073e7f-5d9a-4b80-8170-1ec18c224d7b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:35.353539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d073e7f-5d9a-4b80-8170-1ec18c224d7b None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:35.355149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 598/1201] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:00:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:00:35.360730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9a74e-8b96-4ddf-9d63-b6d007ec9c60 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:35.361333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9a74e-8b96-4ddf-9d63-b6d007ec9c60 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:35.361427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e9a74e-8b96-4ddf-9d63-b6d007ec9c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:35.379887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c1e9a74e-8b96-4ddf-9d63-b6d007ec9c60 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:00:35.381345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c1e9a74e-8b96-4ddf-9d63-b6d007ec9c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:00:35.490201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 604/1202] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:00:35 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:00:35.519641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d073e7f-5d9a-4b80-8170-1ec18c224d7b None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:35.659153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d073e7f-5d9a-4b80-8170-1ec18c224d7b None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:35.703340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:35.704163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:35.704481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:35.705293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:35.717105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:35.718656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:35.738895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:35.740041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:35.741095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:35.741095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:35.741095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:35.741095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:35.741095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 599/1203] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:35.746450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e9a74e-8b96-4ddf-9d63-b6d007ec9c60 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:35.875719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e9a74e-8b96-4ddf-9d63-b6d007ec9c60 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:35.922060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:35.929036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54f1f2d-ccb0-44dd-af67-41c19f94b865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:35.929825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54f1f2d-ccb0-44dd-af67-41c19f94b865 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:35.930109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54f1f2d-ccb0-44dd-af67-41c19f94b865 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:35.930387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54f1f2d-ccb0-44dd-af67-41c19f94b865 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:35.931169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54f1f2d-ccb0-44dd-af67-41c19f94b865 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:35.931580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54f1f2d-ccb0-44dd-af67-41c19f94b865 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:35.933861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54f1f2d-ccb0-44dd-af67-41c19f94b865 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:35.952968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 605/1204] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:35.955258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:36.023821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:36.091124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6612607e-7179-4fad-81f8-3f5a7e731246 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:36.143053 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263328c5-2e7d-489e-99d7-48c188abc140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:36.143924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263328c5-2e7d-489e-99d7-48c188abc140 None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:36.144230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263328c5-2e7d-489e-99d7-48c188abc140 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:36.144559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263328c5-2e7d-489e-99d7-48c188abc140 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:36.145511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263328c5-2e7d-489e-99d7-48c188abc140 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:36.147352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263328c5-2e7d-489e-99d7-48c188abc140 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:36.188418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 600/1205] 10.222.0.20 () {60 vars in 1132 bytes} [Tue Jul 5 10:00:35 2022] POST /identity/v3/users => generated 251 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:00:38.426556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cd30cb-8761-4996-ade0-0333e3ee5f0c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:38.427094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cd30cb-8761-4996-ade0-0333e3ee5f0c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:38.427094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cd30cb-8761-4996-ade0-0333e3ee5f0c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:38.428715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 606/1206] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:00:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:00:38.436046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1b2ef5-f449-4798-99dd-9b9b95f1f807 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:38.436046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1b2ef5-f449-4798-99dd-9b9b95f1f807 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:38.436046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1b2ef5-f449-4798-99dd-9b9b95f1f807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:38.453875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8b1b2ef5-f449-4798-99dd-9b9b95f1f807 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:00:38.455933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8b1b2ef5-f449-4798-99dd-9b9b95f1f807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:00:38.529082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 601/1207] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:00:38 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:00:38.552820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cd30cb-8761-4996-ade0-0333e3ee5f0c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:38.673065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cd30cb-8761-4996-ade0-0333e3ee5f0c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:38.712682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:38.715481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:38.715481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:38.715481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:38.725214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 10:00:38.733021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:38.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:38.735215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 607/1208] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 10:00:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:38.740287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1b2ef5-f449-4798-99dd-9b9b95f1f807 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:38.781553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:38.849581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1b2ef5-f449-4798-99dd-9b9b95f1f807 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:38.888462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:38.889135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:38.889425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:38.889705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:38.890466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:38.890777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:38.892158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:38.905447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 602/1209] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 10:00:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:38.906724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:38.972219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:39.030455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97403318-fb44-49a5-923c-a34ad754ee49 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:39.066340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:39.067137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:39.067442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:39.067706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:39.081281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:39.082762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:39.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:39.102327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:39.103495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:39.103495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:39.103495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:39.103495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:39.104823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 608/1210] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 10:00:39.110405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:39.227451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:39.231321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bef830-ed44-45c7-8f28-615d648b690c None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:39.267793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:39.268679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:39.268964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:39.269264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:39.270052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:39.270397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:39.271769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:39.289942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 603/1211] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:39.291200 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:39.405603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3426dd-6c70-41db-a9d9-89f143bdf06b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:39.407519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:39.452223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:39.453105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:39.453422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:39.453667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] PATH_INFO: `/v3/users/placement` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:39.462622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:39.464171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 10:00:39.481388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:39.482706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:39.483992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:39.485104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 609/1212] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:39 2022] GET /identity/v3/users/placement => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:39.491273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:39.616633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a02c141-5e67-4d65-987e-5c853aedd9d7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:39.662823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:39.663570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:39.663878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:39.664260 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:39.665534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:39.666061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:39.668034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:39.696322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 604/1213] 10.222.0.20 () {56 vars in 1143 bytes} [Tue Jul 5 10:00:39 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:39.703472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:39.839905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd8078-0170-4504-955a-16e0eca2602e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:39.878542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:39.880315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:39.880315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:39.880315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:39.926940 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:39.975149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:39.976771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:40.093352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 610/1214] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 10:00:39 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 10:00:40.099808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:40.120970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:40.217772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a36165-40c4-4020-9e2f-bed83cbec6e5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:40.255601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:40.256434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:40.256751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:40.257029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:40.267871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:40.269348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:40.291837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:40.293040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:40.294156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:40.294156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:40.294156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:40.294156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:40.298473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 605/1215] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:40.299666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:40.371896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:40.427147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c104a4-b3d4-4b19-89e0-bf488e271d3e None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:40.468820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:40.469489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:40.469772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:40.470052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:40.470822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:40.471193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:40.472854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:40.485496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 611/1216] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:40.491209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:40.577308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:40.614862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a207f7-ad76-4720-a75d-753eb0f38465 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:40.654996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:40.655853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:40.656233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:40.656646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:40.664732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:40.666665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 10:00:40.685493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:40.686722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:40.687821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:40.695078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 606/1217] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 10:00:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 10:00:40.697215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:40.819872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ba253e-3841-455b-bd68-ea161f330aa0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:40.843849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:40.856794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e888cf5-52d8-4157-a424-61e76f113ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:40.857462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e888cf5-52d8-4157-a424-61e76f113ddd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:40.857753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e888cf5-52d8-4157-a424-61e76f113ddd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:40.858035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e888cf5-52d8-4157-a424-61e76f113ddd None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:40.858830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e888cf5-52d8-4157-a424-61e76f113ddd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:40.859144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e888cf5-52d8-4157-a424-61e76f113ddd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:40.860619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e888cf5-52d8-4157-a424-61e76f113ddd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:40.876444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 612/1218] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 10:00:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:40.882652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:41.006995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81a7281-f183-464c-a2cf-d35c797fc1fb None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:41.036936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:41.044639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f7a341-898e-4571-a2d0-fccd704f05fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:41.045416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f7a341-898e-4571-a2d0-fccd704f05fd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:41.045770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f7a341-898e-4571-a2d0-fccd704f05fd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:41.046117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f7a341-898e-4571-a2d0-fccd704f05fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:41.047059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f7a341-898e-4571-a2d0-fccd704f05fd None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=947da12bc8304314b5e4a9768b083a7b) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:41.047442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f7a341-898e-4571-a2d0-fccd704f05fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:41.049524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f7a341-898e-4571-a2d0-fccd704f05fd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:41.055371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 607/1219] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 10:00:40 2022] GET /identity/v3/role_assignments?user.id=947da12bc8304314b5e4a9768b083a7b&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:43.180742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561dbe29-d1a3-4a42-b652-10170cd0130a None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:43.181294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561dbe29-d1a3-4a42-b652-10170cd0130a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:43.181487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561dbe29-d1a3-4a42-b652-10170cd0130a None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:43.182987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 613/1220] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:00:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:00:43.188598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213e932-e089-42cb-8e30-2ee19f5eafd0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:43.189138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213e932-e089-42cb-8e30-2ee19f5eafd0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:43.189286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5213e932-e089-42cb-8e30-2ee19f5eafd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:43.207232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5213e932-e089-42cb-8e30-2ee19f5eafd0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:00:43.208820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5213e932-e089-42cb-8e30-2ee19f5eafd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:00:43.282781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 608/1221] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:00:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:00:43.305760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561dbe29-d1a3-4a42-b652-10170cd0130a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:43.411291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561dbe29-d1a3-4a42-b652-10170cd0130a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:43.446919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:43.447804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:43.448072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:43.448354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:43.460686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 10:00:43.471894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:43.473374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:43.474469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 614/1222] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 10:00:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:43.479583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5213e932-e089-42cb-8e30-2ee19f5eafd0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:43.597482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5213e932-e089-42cb-8e30-2ee19f5eafd0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:43.622351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:43.633909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:43.634580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:43.634864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:43.635139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:43.635885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] RBAC: Adding query filter params (name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:43.636192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:43.637613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:43.646777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 609/1223] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 10:00:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:43.654218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:43.706585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:43.766669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f2d91b-07ca-4fa7-b9e2-9995b5ac65c8 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:43.802067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:43.802860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:43.803142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:43.803415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:43.813641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:43.815772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:43.833193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:43.834299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:43.835399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:43.835399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:43.835399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:43.835399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:43.835399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 615/1224] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:43.840305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:43.946051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f64d0e-b471-44c4-8cbe-afe13c56ddb5 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:43.984022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:43.984728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:43.985014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:43.985290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:43.986050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:43.986987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:43.988571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:44.002634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 610/1225] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:44.008649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:44.121242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:44.129649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c069e907-b687-411a-9817-b59e74f62548 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:44.175272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:44.176097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:44.176407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:44.176689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] PATH_INFO: `/v3/users/placement` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:44.184069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:44.185530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 10:00:44.200794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:44.201958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:44.203013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 616/1226] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:44.208244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:44.258386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:44.315395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11952883-7d9f-4ed3-9439-f1063138ccd0 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:44.353130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:44.353850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:44.354164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:44.354552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:44.355429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:44.355748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:44.357521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:44.376489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 611/1227] 10.222.0.20 () {56 vars in 1143 bytes} [Tue Jul 5 10:00:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:44.382532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:44.498486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b9707f-14ee-4445-a8e5-0251e17d017d None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:44.535590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:44.536284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:44.536595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:44.536875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:44.547376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:44.548864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:44.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:44.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:44.576378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:44.576378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:44.576378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:44.576378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:44.576378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 617/1228] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:44.576736 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:44.683854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58547f88-6f77-41b5-95e1-8ab366868ffa None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:44.686302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:44.724845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:44.725514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:44.725825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:44.726098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:44.726878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:44.727286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:44.728913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:44.741376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 612/1229] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:00:44.747171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:44.756224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:44.853891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6a365-8ff6-41e2-944a-293340949ed0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:44.890538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:44.891344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:44.891632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:44.891910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:44.899641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:44.901213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 10:00:44.916981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:44.918137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:44.919246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 618/1230] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 10:00:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:44.924304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:45.031078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3226aeb-dd62-447b-a760-07dd97d90d1e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:45.066986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a2b532-4033-4235-b72f-1c5d1eca0126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:45.067685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2b532-4033-4235-b72f-1c5d1eca0126 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:45.067967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2b532-4033-4235-b72f-1c5d1eca0126 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:45.068241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2b532-4033-4235-b72f-1c5d1eca0126 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:45.069082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2b532-4033-4235-b72f-1c5d1eca0126 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:45.069388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2b532-4033-4235-b72f-1c5d1eca0126 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:45.070830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2b532-4033-4235-b72f-1c5d1eca0126 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:45.084092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 613/1231] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 10:00:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:00:45.090379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:45.163617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:45.202974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa090ee2-50a7-481f-870e-35f4fa7f5fe0 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:45.243348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0099f953-13df-4b00-bff6-4c944cef8535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:45.244202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099f953-13df-4b00-bff6-4c944cef8535 None admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:45.244521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099f953-13df-4b00-bff6-4c944cef8535 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:45.244805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0099f953-13df-4b00-bff6-4c944cef8535 None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/947da12bc8304314b5e4a9768b083a7b/roles/0d7a50b0cac3467b8b2000f706f52329` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:45.260761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0099f953-13df-4b00-bff6-4c944cef8535 None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=947da12bc8304314b5e4a9768b083a7b, role_id=0d7a50b0cac3467b8b2000f706f52329)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:45.262227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0099f953-13df-4b00-bff6-4c944cef8535 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:45.274301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 619/1232] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 10:00:45 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/947da12bc8304314b5e4a9768b083a7b/roles/0d7a50b0cac3467b8b2000f706f52329 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:00:47.401244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f9544-e681-42a9-b38e-b1d1550eeb82 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:47.401797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f9544-e681-42a9-b38e-b1d1550eeb82 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:47.401944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f9544-e681-42a9-b38e-b1d1550eeb82 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:47.403780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 614/1233] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:00:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:00:47.409778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cb7fae-1bb4-4703-9bda-e94f6ec365d9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:47.410245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cb7fae-1bb4-4703-9bda-e94f6ec365d9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:47.410345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cb7fae-1bb4-4703-9bda-e94f6ec365d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:47.429323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f1cb7fae-1bb4-4703-9bda-e94f6ec365d9 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:00:47.430597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f1cb7fae-1bb4-4703-9bda-e94f6ec365d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:00:47.505216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 620/1234] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:00:47 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:00:47.530119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634f9544-e681-42a9-b38e-b1d1550eeb82 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:47.557254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:47.638733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634f9544-e681-42a9-b38e-b1d1550eeb82 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:47.679791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:47.681404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:47.681404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:47.681404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:47.694864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 10:00:47.706153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:47.707351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:47.713248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 615/1235] 10.222.0.20 () {56 vars in 1102 bytes} [Tue Jul 5 10:00:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:47.713977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cb7fae-1bb4-4703-9bda-e94f6ec365d9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:47.764283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:47.825996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cb7fae-1bb4-4703-9bda-e94f6ec365d9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:47.866335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:47.867031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:47.867349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:47.867692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:47.868469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] RBAC: Adding query filter params (name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:47.868779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:47.870340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:47.879189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 621/1236] 10.222.0.20 () {56 vars in 1103 bytes} [Tue Jul 5 10:00:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:47.885073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:47.986181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:48.005167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487444b8-0a45-4925-9d68-bf24372e122a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:48.042149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:48.042946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:48.043230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:48.043506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:48.054701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:48.056146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:48.074511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:48.075603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:48.076724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:48.076724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:48.076724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:48.076724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:48.076724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 616/1237] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:48.082404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:48.199639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd615add-8402-44df-850f-b6513ad4f74a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:48.208761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:48.238763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:48.239390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:48.239680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:48.240009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:48.240884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:48.241245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:48.242775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:48.256416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 622/1238] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:48.262107 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:48.388523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d032e685-bff4-46ae-a8c7-925bfc005f9a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:48.427693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:48.428531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:48.428820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:48.429108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:48.432057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:48.438525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:48.440919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 10:00:48.455673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:48.456903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:48.457865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 617/1239] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:48 2022] GET /identity/v3/users/placement => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:48.462859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:48.566854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f491b-e93d-4ed4-a730-7476b16da3a2 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:48.602928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:48.603635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:48.603921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:48.604200 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:48.605200 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:48.605515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:48.607067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:48.626034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 623/1240] 10.222.0.20 () {56 vars in 1143 bytes} [Tue Jul 5 10:00:48 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:48.631949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:48.757010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb61ca-21d6-4d4f-91b7-efdfcd5e6fa2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:48.799514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:48.800452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:48.800861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:48.801141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:48.869626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:48.877991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:48.879618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:49.004900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 618/1241] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 10:00:48 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 05 10:00:49.006316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:49.035371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:49.109398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6624a5-47f6-4a20-b79d-c5048c71a49c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:49.144666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:49.145391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:49.145678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:49.145996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:49.157272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:49.159431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:49.177431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:49.178505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:49.179457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:49.179457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:49.179457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:49.179457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:49.179457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 624/1242] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:49.184818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:49.291470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:49.299073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f484a946-9dbb-41d0-b564-b3c05065adb7 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:49.338040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:49.339032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:49.339375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:49.339753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:49.340845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:49.341225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:49.342925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:49.356506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 619/1243] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:49.362794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:49.450530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:49.474458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dda1e0-f372-4523-8080-c16508bdd30b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:49.509910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:49.510701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:49.511045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:49.511340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:49.521637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:49.523101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 10:00:49.537638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:49.544821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:49.545736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 625/1244] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 10:00:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:49.546225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:49.659367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fea883-a598-4545-a4e1-36e6a55cd6c0 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:49.702342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bd6f7f-8a6d-439c-8019-08ec655e7a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:49.703005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd6f7f-8a6d-439c-8019-08ec655e7a7a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:49.703295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd6f7f-8a6d-439c-8019-08ec655e7a7a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:49.703604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd6f7f-8a6d-439c-8019-08ec655e7a7a None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:49.704504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bd6f7f-8a6d-439c-8019-08ec655e7a7a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:49.704824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bd6f7f-8a6d-439c-8019-08ec655e7a7a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:49.706383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bd6f7f-8a6d-439c-8019-08ec655e7a7a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:49.726528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 620/1245] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 10:00:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:00:49.727802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:49.839504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2de430f-943d-4364-ad06-00145dce8b2b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:49.875180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c2c231-9c24-4562-a452-df7b5f4477be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:49.875890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c2c231-9c24-4562-a452-df7b5f4477be None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:49.876214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c2c231-9c24-4562-a452-df7b5f4477be None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:49.876524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c2c231-9c24-4562-a452-df7b5f4477be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:49.877897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c2c231-9c24-4562-a452-df7b5f4477be None admin] RBAC: Adding query filter params (role.id=0d7a50b0cac3467b8b2000f706f52329, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=947da12bc8304314b5e4a9768b083a7b) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:49.878212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c2c231-9c24-4562-a452-df7b5f4477be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:49.880612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c2c231-9c24-4562-a452-df7b5f4477be None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:49.888369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 626/1246] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 10:00:49 2022] GET /identity/v3/role_assignments?user.id=947da12bc8304314b5e4a9768b083a7b&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=0d7a50b0cac3467b8b2000f706f52329 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:52.125240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdd7d46-2469-4ae7-88c8-38d3de80ca80 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:52.125809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdd7d46-2469-4ae7-88c8-38d3de80ca80 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:52.125809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdd7d46-2469-4ae7-88c8-38d3de80ca80 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:52.127921 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 621/1247] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:00:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:00:52.133704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23443a9-cb2e-4d0a-8dfb-aecfe6505a03 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:52.134116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23443a9-cb2e-4d0a-8dfb-aecfe6505a03 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:52.134340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23443a9-cb2e-4d0a-8dfb-aecfe6505a03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:52.140036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:52.153744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b23443a9-cb2e-4d0a-8dfb-aecfe6505a03 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:00:52.154935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b23443a9-cb2e-4d0a-8dfb-aecfe6505a03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:00:52.229469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 627/1248] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:00:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:00:52.255835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdd7d46-2469-4ae7-88c8-38d3de80ca80 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:52.274508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:52.373583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cdd7d46-2469-4ae7-88c8-38d3de80ca80 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:52.416824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:52.417767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:52.418137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:52.418452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:52.431177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 10:00:52.440511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:52.441641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:52.442518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 622/1249] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 10:00:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:52.447844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23443a9-cb2e-4d0a-8dfb-aecfe6505a03 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:52.558667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23443a9-cb2e-4d0a-8dfb-aecfe6505a03 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:52.593443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:52.599247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:52.599944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:52.600301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:52.600611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:52.601371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:52.601682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:52.603197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:52.611683 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 628/1250] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 10:00:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:52.617420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:52.706294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:52.731672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72becfd-5c18-462e-bff6-0931a453c692 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:52.767525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:52.768499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:52.768786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:52.769065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:52.780688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:52.782137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:52.800252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:52.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:52.802621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:52.802621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:52.802621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:52.802621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:52.802621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 623/1251] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:52.809030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:52.915063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df648b-1f87-4e01-82e3-c1a45f8ec860 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:52.956893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:52.957756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:52.958109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:52.958487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:52.959364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:52.959782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:52.961306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:52.974714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 629/1252] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:00:52.980565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:53.091035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef986adc-cac4-4912-b87a-2554ef25afde None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:53.117346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:53.125681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:53.126501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:53.126778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:53.127588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:53.135464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:53.136940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 10:00:53.152133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:53.153554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:53.155106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 624/1253] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:52 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 05 10:00:53.159587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:53.216198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:53.261771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c964e-0b3e-4dc2-a5ae-b3f2e1915676 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:53.295896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:53.296596 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:53.296891 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:53.297185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:53.298021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:53.298316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:53.299758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:53.317271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 630/1254] 10.222.0.20 () {56 vars in 1143 bytes} [Tue Jul 5 10:00:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:53.323044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:53.426045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e5f89-32c6-4afd-857f-46cf51a86074 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:53.463747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:53.464420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:53.464705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:53.464987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:53.523393 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:53.537583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:53.539070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:53.640407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 625/1255] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 10:00:53 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 10:00:53.646347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:53.766381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e82399-6a42-45b4-b3ac-f11f2c4ce432 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:53.779757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:53.804995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:53.805836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:53.806088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:53.806358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:53.819216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:53.820709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:53.838319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:53.839507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:53.841666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:53.841666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:53.841666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:53.841666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:53.841666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 631/1256] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:53.845306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:53.949331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:53.963380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b108edf-5b84-4c53-9881-24acb3a22074 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:54.003043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:54.003927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:54.004245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:54.004555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:54.005275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:54.005661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:54.007526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:54.021353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 626/1257] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:54.026919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:54.143993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4a1bda-0536-4979-81f8-c287851cbf48 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:54.180888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:54.181705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:54.181988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:54.182263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:54.189809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:54.191280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 10:00:54.206337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:54.207441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:00:54.208490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 632/1258] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 10:00:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:54.213701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:54.321506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147d3bd6-250b-4153-a097-75253708b216 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:54.356169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27abb2d9-2c9f-48d2-b64b-38cda54615a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:54.356927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27abb2d9-2c9f-48d2-b64b-38cda54615a1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:54.357209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27abb2d9-2c9f-48d2-b64b-38cda54615a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:54.357479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27abb2d9-2c9f-48d2-b64b-38cda54615a1 None admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:54.358273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27abb2d9-2c9f-48d2-b64b-38cda54615a1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:54.358579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27abb2d9-2c9f-48d2-b64b-38cda54615a1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:54.360038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27abb2d9-2c9f-48d2-b64b-38cda54615a1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:54.373408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 627/1259] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 10:00:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:54.379801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:54.407202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:54.484790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d001ac0-7cc8-44ee-af7f-f75d7839366a None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:54.520204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce301b2-5a1d-4a85-9834-e7d1146e61b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:54.520889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce301b2-5a1d-4a85-9834-e7d1146e61b6 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:54.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce301b2-5a1d-4a85-9834-e7d1146e61b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:54.521451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce301b2-5a1d-4a85-9834-e7d1146e61b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:54.522266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce301b2-5a1d-4a85-9834-e7d1146e61b6 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=947da12bc8304314b5e4a9768b083a7b) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:54.522573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce301b2-5a1d-4a85-9834-e7d1146e61b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:54.523997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce301b2-5a1d-4a85-9834-e7d1146e61b6 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:54.529621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 633/1260] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 10:00:54 2022] GET /identity/v3/role_assignments?user.id=947da12bc8304314b5e4a9768b083a7b&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 259 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:56.501693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570297c6-1b03-478c-9a47-e17a9edd0c1d None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:56.502241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570297c6-1b03-478c-9a47-e17a9edd0c1d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:56.502241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570297c6-1b03-478c-9a47-e17a9edd0c1d None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:56.503600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 628/1261] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:00:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:00:56.508688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8986b9-934c-4921-b73d-3fba895c78da None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:56.509068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8986b9-934c-4921-b73d-3fba895c78da None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:56.509248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8986b9-934c-4921-b73d-3fba895c78da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:56.525960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ca8986b9-934c-4921-b73d-3fba895c78da None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:00:56.527130 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ca8986b9-934c-4921-b73d-3fba895c78da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:00:56.590226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 634/1262] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:00:56 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:00:56.612084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570297c6-1b03-478c-9a47-e17a9edd0c1d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:56.622645 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:56.720737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570297c6-1b03-478c-9a47-e17a9edd0c1d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:56.756578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:56.757303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:56.757588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:56.757867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:56.768579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 10:00:56.776249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:56.777366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:56.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 629/1263] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 10:00:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:56.784447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8986b9-934c-4921-b73d-3fba895c78da None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:56.802005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:56.879271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8986b9-934c-4921-b73d-3fba895c78da None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:56.912087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:56.912768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:56.913052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:56.913326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:56.914056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:56.914358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:56.915715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:56.923850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 635/1264] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 10:00:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:56.929335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:56.994732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:57.028630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc60dde-963c-4e4e-9fe7-5934802da481 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:57.060625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:57.061400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:57.061686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:57.061966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:57.072776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:57.074221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:57.091366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:57.092426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:57.093384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:57.093384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:57.093384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:57.093384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:57.093384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 630/1265] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:57.098352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:57.177137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:57.196365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22818f9-1b80-47bc-af76-e10bb0c1736c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:57.228236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:57.228928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:57.229219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:57.229497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:57.230214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:57.230523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:57.231892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:57.244544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 636/1266] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:57.250266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:57.347079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da61365-c4b6-4ba5-9040-0d455f0b68e1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:57.366433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:57.381155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:57.381953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:57.382238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:57.382516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] PATH_INFO: `/v3/users/placement` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:57.390626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:57.392072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 10:00:57.406445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:57.407529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:57.408702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 631/1267] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:57 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:57.413376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:57.511459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0be64a4-27f7-4987-b85b-d94690332c41 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:57.543655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:57.544312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:57.544629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:57.544913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:57.545744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:57.546057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:57.547486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:57.556405 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:57.565540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 637/1268] 10.222.0.20 () {56 vars in 1143 bytes} [Tue Jul 5 10:00:57 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:57.571021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:57.669244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53b0b31-dcf9-4ed5-9874-4ba71927097a None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:57.707706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:57.708532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:57.708813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:57.709083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:57.719308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:57.720765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:00:57.738255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:57.739296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:57.740634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:00:57.740634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:00:57.740634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:00:57.740634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:57.740634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 632/1269] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:00:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:57.745132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:57.839982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8fff8b-e4fb-4141-a792-1fea1603ea98 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:57.873592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:57.874236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:57.874524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:57.874800 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:57.875507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:57.875813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:57.877225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:57.887983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 638/1270] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:00:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:57.893562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:57.925396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:57.991637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0525d3a1-9ba2-401e-931f-de3218b933f9 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:58.023916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:58.024728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:58.025010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:58.025282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:58.032020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:58.034098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 10:00:58.047943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:00:58.049004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:00:58.049960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 633/1271] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 10:00:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:00:58.054848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:58.111212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:58.151556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d192b1c6-c48a-412c-acd4-f0f5757c7dfd None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:58.184023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e9cabe-c2fb-47dd-a4fe-9748b4c58035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:58.184699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e9cabe-c2fb-47dd-a4fe-9748b4c58035 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:58.184987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e9cabe-c2fb-47dd-a4fe-9748b4c58035 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:58.185265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e9cabe-c2fb-47dd-a4fe-9748b4c58035 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:58.186045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e9cabe-c2fb-47dd-a4fe-9748b4c58035 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:00:58.186353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e9cabe-c2fb-47dd-a4fe-9748b4c58035 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:58.187767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e9cabe-c2fb-47dd-a4fe-9748b4c58035 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:58.199362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 639/1272] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 10:00:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:00:58.205093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:00:58.293260 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:00:58.302220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dbe1b8-a998-4ce2-a331-cf0203324715 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:00:58.335377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f321608-a3c5-483d-8bbf-2c682dcdd8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:00:58.336255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f321608-a3c5-483d-8bbf-2c682dcdd8de None admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:00:58.336574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f321608-a3c5-483d-8bbf-2c682dcdd8de None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:00:58.336852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f321608-a3c5-483d-8bbf-2c682dcdd8de None admin] PATH_INFO: `/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/947da12bc8304314b5e4a9768b083a7b/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:00:58.338765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f321608-a3c5-483d-8bbf-2c682dcdd8de None admin] RBAC: Authorizing `identity:create_grant(project_id=9d922cfbef6241f2a2ac0e51e1d62e82, user_id=947da12bc8304314b5e4a9768b083a7b, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:00:58.340185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f321608-a3c5-483d-8bbf-2c682dcdd8de None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:00:58.352051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 634/1273] 10.222.0.20 () {58 vars in 1439 bytes} [Tue Jul 5 10:00:58 2022] PUT /identity/v3/projects/9d922cfbef6241f2a2ac0e51e1d62e82/users/947da12bc8304314b5e4a9768b083a7b/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:01:00.315494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b431d0-6725-4287-b499-df7dddf1bc32 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:00.316010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b431d0-6725-4287-b499-df7dddf1bc32 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:00.316119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b431d0-6725-4287-b499-df7dddf1bc32 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:00.317460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 640/1274] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:01:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:01:00.322357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef48e1-b7cc-4cf2-8c4a-0215aec07c7f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:00.322744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef48e1-b7cc-4cf2-8c4a-0215aec07c7f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:00.322906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef48e1-b7cc-4cf2-8c4a-0215aec07c7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:00.339127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f7ef48e1-b7cc-4cf2-8c4a-0215aec07c7f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:01:00.340294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f7ef48e1-b7cc-4cf2-8c4a-0215aec07c7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:01:00.404876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 635/1275] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:01:00 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:01:00.427603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b431d0-6725-4287-b499-df7dddf1bc32 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:00.524266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b431d0-6725-4287-b499-df7dddf1bc32 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:00.540877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:00.557432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:00.558203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:00.558482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:00.558752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:00.568604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 05 10:01:00.575962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:00.577135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:00.578198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 641/1276] 10.222.0.20 () {56 vars in 1096 bytes} [Tue Jul 5 10:01:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:00.582875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef48e1-b7cc-4cf2-8c4a-0215aec07c7f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:00.661057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:00.685574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef48e1-b7cc-4cf2-8c4a-0215aec07c7f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:00.720069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:00.720757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:00.721539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:00.721812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:00.722550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:00.722853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:00.724218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:00.732008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 636/1277] 10.222.0.20 () {56 vars in 1099 bytes} [Tue Jul 5 10:01:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:00.738275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:00.834991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da43a0e4-523c-4cef-b422-44d46e23916c None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:00.867711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:00.868509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:00.868805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:00.869087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:00.879272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:00.880738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:01:00.897335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:00.898388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:00.899486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:01:00.899486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:01:00.899486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:01:00.899486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:00.899486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 642/1278] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:01:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:00.904256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:01.002085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832551ed-9fba-4b12-8e3c-ab0631e90ff2 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:01.035212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:01.035857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:01.036141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:01.036439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:01.037156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:01.037465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:01.038826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:01.050163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 637/1279] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:01:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:01.055673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:01.086080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:01.153593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a496cf0a-96e3-43d5-927b-a1afa2787058 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:01.185699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:01.186481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:01.186769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:01.187051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:01.194227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:01.195663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 05 10:01:01.210186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:01.211261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:01.212288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 643/1280] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:01:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:01.216952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:01.220634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:01.312721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377ac6f5-b804-4c77-903d-2ce695be84c1 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:01.345517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:01.346158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:01.346441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:01.346714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:01.347532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:01.347837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:01.349270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:01.365588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 638/1281] 10.222.0.20 () {56 vars in 1143 bytes} [Tue Jul 5 10:01:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:01.371126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:01.458975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:01.468120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65af269-dfd2-40b5-b5c6-56f144bc1c09 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:01.499330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:01.499956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:01.500236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:01.500539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:01.565289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:01.566821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:01.649960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:01.660270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 644/1282] 10.222.0.20 () {58 vars in 1282 bytes} [Tue Jul 5 10:01:01 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 05 10:01:01.666128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:01.734047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:01.772379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c57114a-84e0-4e6a-b9a3-9ef2a125b549 None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:01.805674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:01.806457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:01.806743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:01.807019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:01.817475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:01.818910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 05 10:01:01.835832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:01.836887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:01.837857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 05 10:01:01.837857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 05 10:01:01.837857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 05 10:01:01.837857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:01:01.837857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 639/1283] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:01:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:01.842639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:01.943404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f685bb27-38b4-4f75-803e-e4e59731d5ef None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:01.975567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:01.976214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:01.976521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:01.976798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:01.977645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:01.977960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:01.979312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:01.990672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 645/1284] 10.222.0.20 () {56 vars in 1109 bytes} [Tue Jul 5 10:01:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:01.996185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:02.093001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f1192f-c265-4146-bcb5-6c9871a7bffd None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:02.106854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:02.126871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:02.127682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:02.127966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:02.128243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:02.134786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:02.136217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 05 10:01:02.149672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:02.150674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: ERROR keystone.server.flask.application Jul 05 10:01:02.151647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 640/1285] 10.222.0.20 () {56 vars in 1111 bytes} [Tue Jul 5 10:01:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:02.156726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:02.205012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:02.254182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720fc65-a2a8-419f-aa24-c46156be2454 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:02.286439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6c9d4d-1b17-42f3-a9ad-685a46199e63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:02.287082 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c9d4d-1b17-42f3-a9ad-685a46199e63 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:02.287368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c9d4d-1b17-42f3-a9ad-685a46199e63 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:02.287642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6c9d4d-1b17-42f3-a9ad-685a46199e63 None admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:02.288439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6c9d4d-1b17-42f3-a9ad-685a46199e63 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:02.288750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6c9d4d-1b17-42f3-a9ad-685a46199e63 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:02.290157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6c9d4d-1b17-42f3-a9ad-685a46199e63 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:02.302194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 646/1286] 10.222.0.20 () {56 vars in 1148 bytes} [Tue Jul 5 10:01:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:02.308195 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:02.406366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc78e932-1462-4a5f-a97e-d1319953cf4e None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:02.439173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449bc263-281a-4364-a27d-e6ab6f9b8a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:02.439833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bc263-281a-4364-a27d-e6ab6f9b8a69 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:02.440118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bc263-281a-4364-a27d-e6ab6f9b8a69 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:02.440423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449bc263-281a-4364-a27d-e6ab6f9b8a69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:02.441249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449bc263-281a-4364-a27d-e6ab6f9b8a69 None admin] RBAC: Adding query filter params (role.id=1f92a800392649b3b251840ec8dfa2e6, scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82, user.id=947da12bc8304314b5e4a9768b083a7b) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:02.441564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449bc263-281a-4364-a27d-e6ab6f9b8a69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:02.442985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449bc263-281a-4364-a27d-e6ab6f9b8a69 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:02.448666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 641/1287] 10.222.0.20 () {56 vars in 1374 bytes} [Tue Jul 5 10:01:02 2022] GET /identity/v3/role_assignments?user.id=947da12bc8304314b5e4a9768b083a7b&scope.project.id=9d922cfbef6241f2a2ac0e51e1d62e82&role.id=1f92a800392649b3b251840ec8dfa2e6 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:04.445354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5cf0b0-74fd-4f6b-9414-117e742e93e4 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:04.445874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5cf0b0-74fd-4f6b-9414-117e742e93e4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:04.445987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5cf0b0-74fd-4f6b-9414-117e742e93e4 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:04.447272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 647/1288] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:01:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:01:04.452269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f2f45-2753-44f9-bd3e-8616e1a96423 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:04.452662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f2f45-2753-44f9-bd3e-8616e1a96423 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:04.452851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367f2f45-2753-44f9-bd3e-8616e1a96423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:04.469035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-367f2f45-2753-44f9-bd3e-8616e1a96423 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:01:04.470196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-367f2f45-2753-44f9-bd3e-8616e1a96423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:01:04.499299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:04.535299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 642/1289] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:01:04 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 05 10:01:04.557415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5cf0b0-74fd-4f6b-9414-117e742e93e4 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:04.654499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5cf0b0-74fd-4f6b-9414-117e742e93e4 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:04.665305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:04.692235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:04.693246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:04.693597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:04.693933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] PATH_INFO: `/v3/services/placement` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:04.694764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:04.696478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:04.710689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:04.711930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:04.713010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 648/1290] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 10:01:04 2022] GET /identity/v3/services/placement => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:04.718734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367f2f45-2753-44f9-bd3e-8616e1a96423 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:04.817927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367f2f45-2753-44f9-bd3e-8616e1a96423 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:04.850652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e4dfbf-82e9-466a-a22c-a0adb176afb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:04.851298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4dfbf-82e9-466a-a22c-a0adb176afb2 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:04.851580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4dfbf-82e9-466a-a22c-a0adb176afb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:04.851852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e4dfbf-82e9-466a-a22c-a0adb176afb2 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:04.852581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4dfbf-82e9-466a-a22c-a0adb176afb2 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:04.852886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4dfbf-82e9-466a-a22c-a0adb176afb2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:04.854242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e4dfbf-82e9-466a-a22c-a0adb176afb2 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:04.858901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 643/1291] 10.222.0.20 () {56 vars in 1116 bytes} [Tue Jul 5 10:01:04 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:04.864658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:04.963144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35120d14-0ef7-499f-b212-ddff0651aa5b None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:04.994688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17774ae-aa48-428b-ba5f-99c83b84c6a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:04.995334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17774ae-aa48-428b-ba5f-99c83b84c6a7 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:04.995614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17774ae-aa48-428b-ba5f-99c83b84c6a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:04.995889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17774ae-aa48-428b-ba5f-99c83b84c6a7 None admin] PATH_INFO: `/v3/services` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:04.996676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17774ae-aa48-428b-ba5f-99c83b84c6a7 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:04.996982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17774ae-aa48-428b-ba5f-99c83b84c6a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:04.998338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17774ae-aa48-428b-ba5f-99c83b84c6a7 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:05.002657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 649/1292] 10.222.0.20 () {56 vars in 1116 bytes} [Tue Jul 5 10:01:04 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:06.972869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcd433a-bb41-4663-8277-2b1bf126b8af None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:06.973325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcd433a-bb41-4663-8277-2b1bf126b8af None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:06.973428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcd433a-bb41-4663-8277-2b1bf126b8af None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:06.974785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 644/1293] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:01:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:01:06.979946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de3e59-5c59-48c2-b8b5-aa4f75349d3e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:06.980324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de3e59-5c59-48c2-b8b5-aa4f75349d3e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:06.980544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5de3e59-5c59-48c2-b8b5-aa4f75349d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:06.996956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f5de3e59-5c59-48c2-b8b5-aa4f75349d3e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:01:06.998125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f5de3e59-5c59-48c2-b8b5-aa4f75349d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:01:07.026409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:07.062657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 650/1294] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:01:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:01:07.085222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcd433a-bb41-4663-8277-2b1bf126b8af None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:07.115436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:07.183659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcd433a-bb41-4663-8277-2b1bf126b8af None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:07.215908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313814e1-e6f7-4942-bda7-5489d5cc5bde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:07.216578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313814e1-e6f7-4942-bda7-5489d5cc5bde None admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:07.216860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313814e1-e6f7-4942-bda7-5489d5cc5bde None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:07.217133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313814e1-e6f7-4942-bda7-5489d5cc5bde None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:07.217846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313814e1-e6f7-4942-bda7-5489d5cc5bde None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:07.219191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313814e1-e6f7-4942-bda7-5489d5cc5bde None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:07.232792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 645/1295] 10.222.0.20 () {60 vars in 1141 bytes} [Tue Jul 5 10:01:07 2022] POST /identity/v3/services => generated 246 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:01:09.253370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a65399-07e2-41ed-b85d-8e712ef7d58d None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:09.253909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a65399-07e2-41ed-b85d-8e712ef7d58d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:09.253909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a65399-07e2-41ed-b85d-8e712ef7d58d None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:09.255312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 651/1296] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:01:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:01:09.260402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dc7a26-c3ba-42f8-bc19-4370288eca92 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:09.260830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dc7a26-c3ba-42f8-bc19-4370288eca92 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:09.260971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dc7a26-c3ba-42f8-bc19-4370288eca92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:09.277999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-23dc7a26-c3ba-42f8-bc19-4370288eca92 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:01:09.279449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-23dc7a26-c3ba-42f8-bc19-4370288eca92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:01:09.350238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 646/1297] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:01:09 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:01:09.372535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a65399-07e2-41ed-b85d-8e712ef7d58d None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:09.470461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a65399-07e2-41ed-b85d-8e712ef7d58d None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:09.503190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:09.504031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:09.504271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:09.504581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:09.505236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:09.506604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:09.518764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:09.519927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:09.521174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 652/1298] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 10:01:09 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:09.525928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dc7a26-c3ba-42f8-bc19-4370288eca92 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:09.544518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:09.622177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dc7a26-c3ba-42f8-bc19-4370288eca92 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:09.655317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:09.655962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:09.656244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:09.656551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:09.657251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] RBAC: Adding query filter params (name=placement) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:09.657555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:09.658945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:09.663912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 647/1299] 10.222.0.20 () {56 vars in 1116 bytes} [Tue Jul 5 10:01:09 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:09.670784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:09.703251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:09.781507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215762cf-02bb-4c14-a91e-ed147256e972 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:09.813787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9604ae1c-64f3-4fc5-9232-0a87d5e2efd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:09.814450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9604ae1c-64f3-4fc5-9232-0a87d5e2efd8 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:09.814732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9604ae1c-64f3-4fc5-9232-0a87d5e2efd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:09.815520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9604ae1c-64f3-4fc5-9232-0a87d5e2efd8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:09.816299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9604ae1c-64f3-4fc5-9232-0a87d5e2efd8 None admin] RBAC: Adding query filter params (interface=public, service_id=838b8290af63464eab2fa0e4b7ef3bd4, region_id=RegionOne) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:09.816657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9604ae1c-64f3-4fc5-9232-0a87d5e2efd8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:09.818018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9604ae1c-64f3-4fc5-9232-0a87d5e2efd8 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:09.823334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 653/1300] 10.222.0.20 () {56 vars in 1251 bytes} [Tue Jul 5 10:01:09 2022] GET /identity/v3/endpoints?service_id=838b8290af63464eab2fa0e4b7ef3bd4&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:09.828838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:09.912094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:09.932585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ce68c9-93b5-4ff5-b6d2-b068dd25a8ca None admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:09.966816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d945452-0f55-423d-a055-45d129ed7618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:09.967486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d945452-0f55-423d-a055-45d129ed7618 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:09.967772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d945452-0f55-423d-a055-45d129ed7618 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:09.968137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d945452-0f55-423d-a055-45d129ed7618 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:09.968853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d945452-0f55-423d-a055-45d129ed7618 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:09.970231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d945452-0f55-423d-a055-45d129ed7618 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:09.977150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 648/1301] 10.222.0.20 () {56 vars in 1087 bytes} [Tue Jul 5 10:01:09 2022] GET /identity/v3/services => generated 2191 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:01:11.959702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d35f47-67ba-4a3e-8b2c-09b9c0fc3722 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:11.960111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d35f47-67ba-4a3e-8b2c-09b9c0fc3722 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:11.960251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d35f47-67ba-4a3e-8b2c-09b9c0fc3722 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:11.961661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 654/1302] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:01:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:01:11.966664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68adc59f-f97f-4908-b1cb-5359b1581e59 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:11.967045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68adc59f-f97f-4908-b1cb-5359b1581e59 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:11.967231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68adc59f-f97f-4908-b1cb-5359b1581e59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:11.984147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-68adc59f-f97f-4908-b1cb-5359b1581e59 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:01:11.985366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-68adc59f-f97f-4908-b1cb-5359b1581e59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:01:12.050578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 649/1303] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:01:11 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 05 10:01:12.073327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d35f47-67ba-4a3e-8b2c-09b9c0fc3722 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:12.164530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:12.171270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d35f47-67ba-4a3e-8b2c-09b9c0fc3722 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:12.203365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:12.204139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:12.204440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:12.204716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:12.205357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:12.207233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 05 10:01:12.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 05 10:01:12.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.get_or_create( Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application with Lock( Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self._enter() Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application return self.creator() Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application created_value = creator( Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: ERROR keystone.server.flask.application Jul 05 10:01:12.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 655/1304] 10.222.0.20 () {56 vars in 1117 bytes} [Tue Jul 5 10:01:12 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 05 10:01:12.226365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68adc59f-f97f-4908-b1cb-5359b1581e59 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:12.300449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:12.323007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68adc59f-f97f-4908-b1cb-5359b1581e59 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:12.356381 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e42435-1b35-42cf-8fcc-16ea38655690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:12.357062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e42435-1b35-42cf-8fcc-16ea38655690 None admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:12.357351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e42435-1b35-42cf-8fcc-16ea38655690 None admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:12.357629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e42435-1b35-42cf-8fcc-16ea38655690 None admin] PATH_INFO: `/v3/services` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:12.358330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e42435-1b35-42cf-8fcc-16ea38655690 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:01:12.358638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e42435-1b35-42cf-8fcc-16ea38655690 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:12.360012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e42435-1b35-42cf-8fcc-16ea38655690 None admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:12.365124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 650/1305] 10.222.0.20 () {56 vars in 1116 bytes} [Tue Jul 5 10:01:12 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:01:12.370650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:01:12.467331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf933891-1006-456e-be25-eb73a554db80 None admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:01:12.499314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4339e6-e11c-410c-b4b3-afff14a53cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:01:12.499943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4339e6-e11c-410c-b4b3-afff14a53cdc None admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:01:12.500226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4339e6-e11c-410c-b4b3-afff14a53cdc None admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:01:12.500532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4339e6-e11c-410c-b4b3-afff14a53cdc None admin] PATH_INFO: `/v3/endpoints` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:01:12.501269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4339e6-e11c-410c-b4b3-afff14a53cdc None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:01:12.502642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4339e6-e11c-410c-b4b3-afff14a53cdc None admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:01:12.520098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 656/1306] 10.222.0.20 () {60 vars in 1144 bytes} [Tue Jul 5 10:01:12 2022] POST /identity/v3/endpoints => generated 331 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:01:22.362227 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=30 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:01:22.516487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:14.456261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c9525-9b18-4b06-b02d-a1dfe522ec94 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:14.456840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c9525-9b18-4b06-b02d-a1dfe522ec94 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:14.456932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c9525-9b18-4b06-b02d-a1dfe522ec94 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:14.462129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 651/1307] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:14 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:14.463506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1209fdd6-b7d7-4095-94b1-e79921147968 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:14.463936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1209fdd6-b7d7-4095-94b1-e79921147968 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:14.464140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1209fdd6-b7d7-4095-94b1-e79921147968 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:14.549803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1209fdd6-b7d7-4095-94b1-e79921147968 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:14.551397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1209fdd6-b7d7-4095-94b1-e79921147968 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:14.743640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 657/1308] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:02:14.799544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ce50c-f966-4e56-a680-d8cb747abc31 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:14.799544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ce50c-f966-4e56-a680-d8cb747abc31 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:14.799544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014ce50c-f966-4e56-a680-d8cb747abc31 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:14.799544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 652/1309] 10.222.0.20 () {54 vars in 964 bytes} [Tue Jul 5 10:02:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:14.810109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195dedc-537a-416c-9e1c-d49d8aebd424 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:14.810409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195dedc-537a-416c-9e1c-d49d8aebd424 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:14.810686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d195dedc-537a-416c-9e1c-d49d8aebd424 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:14.907914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d195dedc-537a-416c-9e1c-d49d8aebd424 None None] MFA Rules not processed for user `36ce9bd8ce6448b8b09fe9eb7b8f7c54`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:14.909667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d195dedc-537a-416c-9e1c-d49d8aebd424 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:14.934825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:14.996552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 658/1310] 10.222.0.20 () {58 vars in 1061 bytes} [Tue Jul 5 10:02:14 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:15.004303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014ce50c-f966-4e56-a680-d8cb747abc31 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:15.006092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-014ce50c-f966-4e56-a680-d8cb747abc31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:15.120032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014ce50c-f966-4e56-a680-d8cb747abc31 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:15.198609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfba4b4b-01ad-4662-b9c6-51d185679a81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:15.199267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba4b4b-01ad-4662-b9c6-51d185679a81 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:15.199740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba4b4b-01ad-4662-b9c6-51d185679a81 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:15.200026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba4b4b-01ad-4662-b9c6-51d185679a81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:15.201708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bfba4b4b-01ad-4662-b9c6-51d185679a81 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:15.260979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:15.319074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfba4b4b-01ad-4662-b9c6-51d185679a81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:15.320543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfba4b4b-01ad-4662-b9c6-51d185679a81 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:15.340323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 653/1311] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:17.665387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed34e9-0d7e-4a88-b5aa-9f7b3de41d31 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:17.665942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed34e9-0d7e-4a88-b5aa-9f7b3de41d31 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:17.665942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ed34e9-0d7e-4a88-b5aa-9f7b3de41d31 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:17.667338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 659/1312] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:17.672495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e781ef-b93c-4280-97ad-3ca189b322b1 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:17.672923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e781ef-b93c-4280-97ad-3ca189b322b1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:17.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e781ef-b93c-4280-97ad-3ca189b322b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:17.743969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f1e781ef-b93c-4280-97ad-3ca189b322b1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:17.745199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f1e781ef-b93c-4280-97ad-3ca189b322b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:17.843300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 654/1313] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:17.881314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ed34e9-0d7e-4a88-b5aa-9f7b3de41d31 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:17.882855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-44ed34e9-0d7e-4a88-b5aa-9f7b3de41d31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:18.008672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ed34e9-0d7e-4a88-b5aa-9f7b3de41d31 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:18.091209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7712d8ec-3065-4a25-b7f6-b7e159816772 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:18.091913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7712d8ec-3065-4a25-b7f6-b7e159816772 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:18.092236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7712d8ec-3065-4a25-b7f6-b7e159816772 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:18.092458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7712d8ec-3065-4a25-b7f6-b7e159816772 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:18.094074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7712d8ec-3065-4a25-b7f6-b7e159816772 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:18.096588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:18.210666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7712d8ec-3065-4a25-b7f6-b7e159816772 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:18.212109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7712d8ec-3065-4a25-b7f6-b7e159816772 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:18.233292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 660/1314] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:20.591417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce7bf80-6c2e-45d5-9300-29ca7605b7e0 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:20.591938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce7bf80-6c2e-45d5-9300-29ca7605b7e0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:20.592068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce7bf80-6c2e-45d5-9300-29ca7605b7e0 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:20.593389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 655/1315] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:20.599294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbecf23-0f61-4880-83d5-a658902e217e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:20.599664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbecf23-0f61-4880-83d5-a658902e217e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:20.599850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbecf23-0f61-4880-83d5-a658902e217e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:20.665484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5bbecf23-0f61-4880-83d5-a658902e217e None None] MFA Rules not processed for user `02951879c7ac4670a6f8f5177df11cb3`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:20.666627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5bbecf23-0f61-4880-83d5-a658902e217e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:20.718002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:20.766199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 661/1316] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:20 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:20.805978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce7bf80-6c2e-45d5-9300-29ca7605b7e0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:20.807508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7ce7bf80-6c2e-45d5-9300-29ca7605b7e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:20.817269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:20.910157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce7bf80-6c2e-45d5-9300-29ca7605b7e0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:20.984096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d78255-202b-4c2c-96f5-b7e82ba3ef0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:20.984789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d78255-202b-4c2c-96f5-b7e82ba3ef0b service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:20.985073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d78255-202b-4c2c-96f5-b7e82ba3ef0b service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:20.985352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d78255-202b-4c2c-96f5-b7e82ba3ef0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:20.986985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-60d78255-202b-4c2c-96f5-b7e82ba3ef0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:21.047558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:21.094086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d78255-202b-4c2c-96f5-b7e82ba3ef0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:21.095511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d78255-202b-4c2c-96f5-b7e82ba3ef0b service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:21.114513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 656/1317] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:20 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:23.637236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb9926-8e51-4abf-874e-188e1847673c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:23.637796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb9926-8e51-4abf-874e-188e1847673c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:23.637796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb9926-8e51-4abf-874e-188e1847673c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:23.639188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 662/1318] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:23.644333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dfe23c-d46a-43b3-8c45-87abcc0a018e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:23.644742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dfe23c-d46a-43b3-8c45-87abcc0a018e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:23.644934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62dfe23c-d46a-43b3-8c45-87abcc0a018e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:23.708903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-62dfe23c-d46a-43b3-8c45-87abcc0a018e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:23.710120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-62dfe23c-d46a-43b3-8c45-87abcc0a018e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:23.803505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 657/1319] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:23 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:23.841448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb9926-8e51-4abf-874e-188e1847673c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:23.842987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-50fb9926-8e51-4abf-874e-188e1847673c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:23.945559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb9926-8e51-4abf-874e-188e1847673c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:24.015294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652111e8-d3aa-478e-9d4c-cab355112865 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:24.015933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652111e8-d3aa-478e-9d4c-cab355112865 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:24.016214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652111e8-d3aa-478e-9d4c-cab355112865 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:24.016522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652111e8-d3aa-478e-9d4c-cab355112865 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:24.018139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-652111e8-d3aa-478e-9d4c-cab355112865 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:24.025321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:24.123160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652111e8-d3aa-478e-9d4c-cab355112865 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:24.124615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652111e8-d3aa-478e-9d4c-cab355112865 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:24.142452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 663/1320] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:23 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:26.818978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ad7f8-1a5e-44d4-9e80-f57c4e27b74a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:26.819536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ad7f8-1a5e-44d4-9e80-f57c4e27b74a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:26.819536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150ad7f8-1a5e-44d4-9e80-f57c4e27b74a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:26.821003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 658/1321] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:26.826225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e328645f-81a2-4502-b17a-7486aec31035 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:26.826642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e328645f-81a2-4502-b17a-7486aec31035 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:26.826787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e328645f-81a2-4502-b17a-7486aec31035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:26.891687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e328645f-81a2-4502-b17a-7486aec31035 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:26.892917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e328645f-81a2-4502-b17a-7486aec31035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:26.952509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:26.995859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 664/1322] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:27.031604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ad7f8-1a5e-44d4-9e80-f57c4e27b74a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:27.033226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-150ad7f8-1a5e-44d4-9e80-f57c4e27b74a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:27.054715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:27.143819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150ad7f8-1a5e-44d4-9e80-f57c4e27b74a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:27.219002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2196f96-d575-413b-b9aa-6d940261349c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:27.219654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2196f96-d575-413b-b9aa-6d940261349c service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:27.219939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2196f96-d575-413b-b9aa-6d940261349c service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:27.220219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2196f96-d575-413b-b9aa-6d940261349c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:27.221894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c2196f96-d575-413b-b9aa-6d940261349c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:27.294683 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:27.333228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2196f96-d575-413b-b9aa-6d940261349c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:27.334678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2196f96-d575-413b-b9aa-6d940261349c service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:27.353069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 659/1323] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:29.410054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a063101-57ae-486a-9df1-d74fa7641d3c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:29.410560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a063101-57ae-486a-9df1-d74fa7641d3c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:29.410560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a063101-57ae-486a-9df1-d74fa7641d3c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:29.412610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 665/1324] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:29.417854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ee3042-8242-4f2c-af58-db9d14d920b6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:29.418275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ee3042-8242-4f2c-af58-db9d14d920b6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:29.418413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ee3042-8242-4f2c-af58-db9d14d920b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:29.484547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-00ee3042-8242-4f2c-af58-db9d14d920b6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:29.485763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-00ee3042-8242-4f2c-af58-db9d14d920b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:29.584402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 660/1325] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:29.634848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ac70d0-4578-4b9a-be92-e26de6752b3c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:29.635241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ac70d0-4578-4b9a-be92-e26de6752b3c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:29.635578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ac70d0-4578-4b9a-be92-e26de6752b3c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:29.637074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 666/1326] 10.222.0.20 () {54 vars in 964 bytes} [Tue Jul 5 10:02:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:29.643207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ddd3a-92fb-48ef-a0fe-5c8e5ca5e78e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:29.643494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ddd3a-92fb-48ef-a0fe-5c8e5ca5e78e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:29.643768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1ddd3a-92fb-48ef-a0fe-5c8e5ca5e78e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:29.675902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:29.727272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6b1ddd3a-92fb-48ef-a0fe-5c8e5ca5e78e None None] MFA Rules not processed for user `36ce9bd8ce6448b8b09fe9eb7b8f7c54`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:29.728506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6b1ddd3a-92fb-48ef-a0fe-5c8e5ca5e78e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:29.805754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 661/1327] 10.222.0.20 () {58 vars in 1061 bytes} [Tue Jul 5 10:02:29 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:02:29.815047 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ac70d0-4578-4b9a-be92-e26de6752b3c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:29.816751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-03ac70d0-4578-4b9a-be92-e26de6752b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:29.921844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ac70d0-4578-4b9a-be92-e26de6752b3c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:30.001370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10025312-3d0b-4766-8b74-23a52985d554 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:30.002026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10025312-3d0b-4766-8b74-23a52985d554 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:30.002358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10025312-3d0b-4766-8b74-23a52985d554 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:30.002638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10025312-3d0b-4766-8b74-23a52985d554 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:30.004632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-10025312-3d0b-4766-8b74-23a52985d554 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:30.023090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:30.116556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10025312-3d0b-4766-8b74-23a52985d554 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:30.118004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10025312-3d0b-4766-8b74-23a52985d554 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:30.137097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 667/1328] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:32.429666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc28c51-0b7b-4d7f-89d9-7bc7115b9c31 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:32.430124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc28c51-0b7b-4d7f-89d9-7bc7115b9c31 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:32.430213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc28c51-0b7b-4d7f-89d9-7bc7115b9c31 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:32.431598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 662/1329] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:32.436734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ce79c-302a-444d-9fc0-4cb727066fd2 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:32.437162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ce79c-302a-444d-9fc0-4cb727066fd2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:32.437417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ce79c-302a-444d-9fc0-4cb727066fd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:32.500505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-4d4ce79c-302a-444d-9fc0-4cb727066fd2 None None] MFA Rules not processed for user `02951879c7ac4670a6f8f5177df11cb3`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:32.501701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4d4ce79c-302a-444d-9fc0-4cb727066fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:32.575161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:32.614594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 668/1330] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:32 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:02:32.652250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc28c51-0b7b-4d7f-89d9-7bc7115b9c31 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:32.653818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fdc28c51-0b7b-4d7f-89d9-7bc7115b9c31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:32.760171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc28c51-0b7b-4d7f-89d9-7bc7115b9c31 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:32.779359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:32.833091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f99dbd-10bb-454e-aba2-21215e0a61e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:32.833735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f99dbd-10bb-454e-aba2-21215e0a61e0 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:32.834015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f99dbd-10bb-454e-aba2-21215e0a61e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:32.834292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f99dbd-10bb-454e-aba2-21215e0a61e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:32.835910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-25f99dbd-10bb-454e-aba2-21215e0a61e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:32.944369 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f99dbd-10bb-454e-aba2-21215e0a61e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:32.945857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f99dbd-10bb-454e-aba2-21215e0a61e0 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:32.965417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 663/1331] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:32 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:35.222350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40cfe73-4228-4674-907f-6fe76a64dcdc None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:35.222885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40cfe73-4228-4674-907f-6fe76a64dcdc None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:35.222982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40cfe73-4228-4674-907f-6fe76a64dcdc None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:35.224872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 669/1332] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:35.230088 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb9d7ce-e772-4b96-a1f6-5daa10bc9eeb None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:35.230487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb9d7ce-e772-4b96-a1f6-5daa10bc9eeb None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:35.230652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb9d7ce-e772-4b96-a1f6-5daa10bc9eeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:35.292626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:35.295521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-1cb9d7ce-e772-4b96-a1f6-5daa10bc9eeb None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:35.297290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1cb9d7ce-e772-4b96-a1f6-5daa10bc9eeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:35.390604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 664/1333] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:35.426418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40cfe73-4228-4674-907f-6fe76a64dcdc None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:35.428582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c40cfe73-4228-4674-907f-6fe76a64dcdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:35.528046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40cfe73-4228-4674-907f-6fe76a64dcdc None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:35.600265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04189bad-6105-44c8-a5c5-99ffbce2e6d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:35.600926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04189bad-6105-44c8-a5c5-99ffbce2e6d0 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:35.601210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04189bad-6105-44c8-a5c5-99ffbce2e6d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:35.601482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04189bad-6105-44c8-a5c5-99ffbce2e6d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:35.603071 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-04189bad-6105-44c8-a5c5-99ffbce2e6d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:35.628475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:35.710661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04189bad-6105-44c8-a5c5-99ffbce2e6d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:35.712101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04189bad-6105-44c8-a5c5-99ffbce2e6d0 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:35.730949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 670/1334] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:37.821061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1087134-17b0-4a74-85a7-9306b1d40b34 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:37.821620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1087134-17b0-4a74-85a7-9306b1d40b34 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:37.821620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1087134-17b0-4a74-85a7-9306b1d40b34 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:37.822998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 665/1335] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:37.828185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f0fa0-228b-46e6-a829-48a098adf24f None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:37.828648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f0fa0-228b-46e6-a829-48a098adf24f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:37.828779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f0fa0-228b-46e6-a829-48a098adf24f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:37.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-387f0fa0-228b-46e6-a829-48a098adf24f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:37.892593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-387f0fa0-228b-46e6-a829-48a098adf24f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:37.967644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:37.989593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 671/1336] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:38.029450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1087134-17b0-4a74-85a7-9306b1d40b34 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:38.030993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b1087134-17b0-4a74-85a7-9306b1d40b34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:38.133117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1087134-17b0-4a74-85a7-9306b1d40b34 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:38.160948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:38.204796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb502203-f699-423e-bea4-db4150c7a93f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:38.205420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb502203-f699-423e-bea4-db4150c7a93f service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:38.205701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb502203-f699-423e-bea4-db4150c7a93f service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:38.205977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb502203-f699-423e-bea4-db4150c7a93f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:38.207588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-cb502203-f699-423e-bea4-db4150c7a93f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:38.314895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb502203-f699-423e-bea4-db4150c7a93f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:38.316345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb502203-f699-423e-bea4-db4150c7a93f service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:38.335118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 666/1337] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:40.647865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf125665-5f5c-46fe-96c0-c75c59cce2d1 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:40.648383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf125665-5f5c-46fe-96c0-c75c59cce2d1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:40.648383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf125665-5f5c-46fe-96c0-c75c59cce2d1 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:40.649784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 672/1338] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:40.655019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ebe326-a5a6-4160-896e-bc8d8c34f85f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:40.655438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ebe326-a5a6-4160-896e-bc8d8c34f85f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:40.655585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ebe326-a5a6-4160-896e-bc8d8c34f85f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:40.719975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b8ebe326-a5a6-4160-896e-bc8d8c34f85f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:40.721177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b8ebe326-a5a6-4160-896e-bc8d8c34f85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:40.729214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:40.816509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 667/1339] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:40 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:40.851863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf125665-5f5c-46fe-96c0-c75c59cce2d1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:40.853411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bf125665-5f5c-46fe-96c0-c75c59cce2d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:40.955120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf125665-5f5c-46fe-96c0-c75c59cce2d1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:41.026291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa64ae9-32bd-4626-a9f0-82006bfa0108 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:41.026930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa64ae9-32bd-4626-a9f0-82006bfa0108 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:41.027211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa64ae9-32bd-4626-a9f0-82006bfa0108 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:41.027483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa64ae9-32bd-4626-a9f0-82006bfa0108 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:41.029141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-afa64ae9-32bd-4626-a9f0-82006bfa0108 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:41.064343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:41.136036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa64ae9-32bd-4626-a9f0-82006bfa0108 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:41.137493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa64ae9-32bd-4626-a9f0-82006bfa0108 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:41.155143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 673/1340] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:40 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:45.021142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd86ab-82da-4b40-8468-4eba36394b37 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:45.021681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd86ab-82da-4b40-8468-4eba36394b37 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:45.021681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd86ab-82da-4b40-8468-4eba36394b37 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:45.023085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 668/1341] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:45.028259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eace9c-af13-46d9-ab5f-d641db9975d5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:45.028723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eace9c-af13-46d9-ab5f-d641db9975d5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:45.028849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40eace9c-af13-46d9-ab5f-d641db9975d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:45.098256 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-40eace9c-af13-46d9-ab5f-d641db9975d5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:45.099976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-40eace9c-af13-46d9-ab5f-d641db9975d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:45.193868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:45.208424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 674/1342] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:45.245494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd86ab-82da-4b40-8468-4eba36394b37 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:45.247595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8fbd86ab-82da-4b40-8468-4eba36394b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:45.341398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd86ab-82da-4b40-8468-4eba36394b37 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:45.344952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588551bd-8750-41b9-aaef-046fb4c81910 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:45.345674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588551bd-8750-41b9-aaef-046fb4c81910 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:45.346018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588551bd-8750-41b9-aaef-046fb4c81910 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:45.346360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588551bd-8750-41b9-aaef-046fb4c81910 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:45.356756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588551bd-8750-41b9-aaef-046fb4c81910 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:45.358187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588551bd-8750-41b9-aaef-046fb4c81910 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:45.367694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 669/1343] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:48.030689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3294d-8d63-4f0f-a296-62f101744453 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:48.031253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3294d-8d63-4f0f-a296-62f101744453 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:48.031253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3294d-8d63-4f0f-a296-62f101744453 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:48.032837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 675/1344] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:48.038680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bbc408-0706-4b45-a8f2-5ebfcbe1b6a6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:48.039068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bbc408-0706-4b45-a8f2-5ebfcbe1b6a6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:48.039362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bbc408-0706-4b45-a8f2-5ebfcbe1b6a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:48.067188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f7bbc408-0706-4b45-a8f2-5ebfcbe1b6a6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:48.068419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f7bbc408-0706-4b45-a8f2-5ebfcbe1b6a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:48.095570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:02:48.151501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 670/1345] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:48 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:48.189942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d3294d-8d63-4f0f-a296-62f101744453 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:48.200812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d3294d-8d63-4f0f-a296-62f101744453 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:48.204418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0f2003-4837-42cb-b0e4-17bdae124e2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:48.205218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f2003-4837-42cb-b0e4-17bdae124e2b service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:48.205502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f2003-4837-42cb-b0e4-17bdae124e2b service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:48.205773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f2003-4837-42cb-b0e4-17bdae124e2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:48.216136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0f2003-4837-42cb-b0e4-17bdae124e2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:48.217621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb0f2003-4837-42cb-b0e4-17bdae124e2b service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:48.228538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 676/1346] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:48 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:52.153231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd26f1-eed1-4e6a-b981-e6f2477363c4 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:52.153813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd26f1-eed1-4e6a-b981-e6f2477363c4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:52.153813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57bd26f1-eed1-4e6a-b981-e6f2477363c4 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:52.155276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 671/1347] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:02:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:52.160331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b8dfed-df89-49af-be99-02a4bcaeddd8 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:52.160795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b8dfed-df89-49af-be99-02a4bcaeddd8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:52.161054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b8dfed-df89-49af-be99-02a4bcaeddd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:52.179934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-67b8dfed-df89-49af-be99-02a4bcaeddd8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:52.181241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-67b8dfed-df89-49af-be99-02a4bcaeddd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:52.189325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 677/1348] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:02:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:02:52.226732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bd26f1-eed1-4e6a-b981-e6f2477363c4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:02:52.237469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57bd26f1-eed1-4e6a-b981-e6f2477363c4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:02:52.241576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c897ec-9b14-4a84-b437-57cc353f165d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:02:52.242309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c897ec-9b14-4a84-b437-57cc353f165d service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:52.242661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c897ec-9b14-4a84-b437-57cc353f165d service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:52.243005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c897ec-9b14-4a84-b437-57cc353f165d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:52.252599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c897ec-9b14-4a84-b437-57cc353f165d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:02:52.254042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c897ec-9b14-4a84-b437-57cc353f165d service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:02:52.265282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 672/1349] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:02:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:02:57.240192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f763e89-d37c-4cb4-89dc-64c063e62a4c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:57.240192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f763e89-d37c-4cb4-89dc-64c063e62a4c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:57.240192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f763e89-d37c-4cb4-89dc-64c063e62a4c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:57.244772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 678/1350] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:02:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:02:57.246806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efda8f4-1a6d-4262-96b8-887c3715bd04 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:02:57.247548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efda8f4-1a6d-4262-96b8-887c3715bd04 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:02:57.247975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efda8f4-1a6d-4262-96b8-887c3715bd04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:02:57.359048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0efda8f4-1a6d-4262-96b8-887c3715bd04 None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:02:57.360830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0efda8f4-1a6d-4262-96b8-887c3715bd04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:02:57.456882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 673/1351] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:02:57 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:01.577550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87295b40-2826-4fc9-b7ff-0d8f6198f4b8 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:01.578618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87295b40-2826-4fc9-b7ff-0d8f6198f4b8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:01.578986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87295b40-2826-4fc9-b7ff-0d8f6198f4b8 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:01.580640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 679/1352] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:03:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:01.585680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca06d3db-371e-4024-8fdf-05f828f393d6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:01.586123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca06d3db-371e-4024-8fdf-05f828f393d6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:01.586244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca06d3db-371e-4024-8fdf-05f828f393d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:01.605551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ca06d3db-371e-4024-8fdf-05f828f393d6 None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:01.606750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ca06d3db-371e-4024-8fdf-05f828f393d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:01.615305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 674/1353] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:03:01 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:02.176834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=13 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:03:03.111609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbee3ca-2d79-4843-bd1c-664c0ae84fba None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:03.112259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbee3ca-2d79-4843-bd1c-664c0ae84fba None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:03.112259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbee3ca-2d79-4843-bd1c-664c0ae84fba None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:03.113753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 680/1354] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:03:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:03.119093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedeaa20-9bfd-46a6-9a64-683c7f166cc1 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:03.119476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedeaa20-9bfd-46a6-9a64-683c7f166cc1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:03.119679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedeaa20-9bfd-46a6-9a64-683c7f166cc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:03.137730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-eedeaa20-9bfd-46a6-9a64-683c7f166cc1 None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:03.138993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-eedeaa20-9bfd-46a6-9a64-683c7f166cc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:03.147620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 675/1355] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:03:03 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:03:06.722966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ca678-5c19-4ac7-b8fd-18ba7b7500e8 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:06.723515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ca678-5c19-4ac7-b8fd-18ba7b7500e8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:06.723515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ca678-5c19-4ac7-b8fd-18ba7b7500e8 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:06.725086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 681/1356] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:06.730545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628014a6-7d67-4844-b793-c216437a90cc None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:06.730986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628014a6-7d67-4844-b793-c216437a90cc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:06.731081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628014a6-7d67-4844-b793-c216437a90cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:06.749229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-628014a6-7d67-4844-b793-c216437a90cc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:06.750413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-628014a6-7d67-4844-b793-c216437a90cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:06.758429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 676/1357] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:06.814681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268d24b-8b1f-4bd2-bfd0-8b0e68e70cbb None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:06.814981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268d24b-8b1f-4bd2-bfd0-8b0e68e70cbb None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:06.815267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268d24b-8b1f-4bd2-bfd0-8b0e68e70cbb None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:06.816944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 682/1358] 10.222.0.20 () {54 vars in 961 bytes} [Tue Jul 5 10:03:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:06.827300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f111c167-1570-4bb1-a2f4-383024bc5131 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:06.827941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f111c167-1570-4bb1-a2f4-383024bc5131 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:06.828406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f111c167-1570-4bb1-a2f4-383024bc5131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:06.876188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:03:06.884874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f111c167-1570-4bb1-a2f4-383024bc5131 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:06.886063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f111c167-1570-4bb1-a2f4-383024bc5131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:06.981824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 677/1359] 10.222.0.20 () {58 vars in 1058 bytes} [Tue Jul 5 10:03:06 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:06.989671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268d24b-8b1f-4bd2-bfd0-8b0e68e70cbb None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:07.004699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268d24b-8b1f-4bd2-bfd0-8b0e68e70cbb None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:07.010188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e95bf1e-d76d-4170-910f-914acf622f37 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:07.010836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95bf1e-d76d-4170-910f-914acf622f37 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:07.011125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95bf1e-d76d-4170-910f-914acf622f37 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:07.011407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e95bf1e-d76d-4170-910f-914acf622f37 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:07.020235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e95bf1e-d76d-4170-910f-914acf622f37 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:07.021924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e95bf1e-d76d-4170-910f-914acf622f37 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:07.032745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 683/1360] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:09.268332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64714967-130d-440d-8c26-04e886f73f13 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:09.268858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64714967-130d-440d-8c26-04e886f73f13 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:09.268966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64714967-130d-440d-8c26-04e886f73f13 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:09.270574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 678/1361] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:09.276078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8850c9b8-8dcd-4579-a7dd-df55001df1d3 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:09.276505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8850c9b8-8dcd-4579-a7dd-df55001df1d3 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:09.276690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8850c9b8-8dcd-4579-a7dd-df55001df1d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:09.295487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8850c9b8-8dcd-4579-a7dd-df55001df1d3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:09.296717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8850c9b8-8dcd-4579-a7dd-df55001df1d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:09.304814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 684/1362] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:09.348671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64714967-130d-440d-8c26-04e886f73f13 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:09.364359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64714967-130d-440d-8c26-04e886f73f13 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:09.369768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaa5535-cdfc-4bf1-98f7-1d2aea84d608 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:09.370411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaa5535-cdfc-4bf1-98f7-1d2aea84d608 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:09.370696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaa5535-cdfc-4bf1-98f7-1d2aea84d608 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:09.370978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaa5535-cdfc-4bf1-98f7-1d2aea84d608 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:09.379830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaa5535-cdfc-4bf1-98f7-1d2aea84d608 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:09.381838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eaa5535-cdfc-4bf1-98f7-1d2aea84d608 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:09.392178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 679/1363] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:11.732276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af7ce49-8dd5-41a0-89cc-0728b0b3526f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:11.732794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af7ce49-8dd5-41a0-89cc-0728b0b3526f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:11.732890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af7ce49-8dd5-41a0-89cc-0728b0b3526f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:11.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 685/1364] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:11.739558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094dad11-ffd6-4c9e-97d1-1e05f149864d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:11.739990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094dad11-ffd6-4c9e-97d1-1e05f149864d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:11.740157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094dad11-ffd6-4c9e-97d1-1e05f149864d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:11.762829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-094dad11-ffd6-4c9e-97d1-1e05f149864d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:11.763491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-094dad11-ffd6-4c9e-97d1-1e05f149864d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:11.773670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 680/1365] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:11.826368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14581649-4b1c-4524-a340-0eaed86b5eb8 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:11.826656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14581649-4b1c-4524-a340-0eaed86b5eb8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:11.826936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14581649-4b1c-4524-a340-0eaed86b5eb8 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:11.828308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 686/1366] 10.222.0.20 () {54 vars in 961 bytes} [Tue Jul 5 10:03:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:11.833853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a0d0de-05d2-4389-bfea-d6117a9a5b95 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:11.834148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a0d0de-05d2-4389-bfea-d6117a9a5b95 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:11.834431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a0d0de-05d2-4389-bfea-d6117a9a5b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:11.853305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-54a0d0de-05d2-4389-bfea-d6117a9a5b95 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:11.854499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-54a0d0de-05d2-4389-bfea-d6117a9a5b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:11.863756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 681/1367] 10.222.0.20 () {58 vars in 1058 bytes} [Tue Jul 5 10:03:11 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:11.872073 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14581649-4b1c-4524-a340-0eaed86b5eb8 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:11.886172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14581649-4b1c-4524-a340-0eaed86b5eb8 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:11.891691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c947b7-989f-4e93-b096-1f797e5bfde9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:11.892331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c947b7-989f-4e93-b096-1f797e5bfde9 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:11.892685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c947b7-989f-4e93-b096-1f797e5bfde9 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:11.892968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c947b7-989f-4e93-b096-1f797e5bfde9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:11.901408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c947b7-989f-4e93-b096-1f797e5bfde9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:11.903095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c947b7-989f-4e93-b096-1f797e5bfde9 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:11.912483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 687/1368] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:13.998681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc333c7-97e3-4253-be32-d7557a60a36e None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:13.999220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc333c7-97e3-4253-be32-d7557a60a36e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:13.999220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc333c7-97e3-4253-be32-d7557a60a36e None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:14.000625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 682/1369] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:14.005799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1880d-240a-4398-ae0f-ae6205d6134a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:14.006713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1880d-240a-4398-ae0f-ae6205d6134a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:14.007037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc1880d-240a-4398-ae0f-ae6205d6134a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:14.025235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0dc1880d-240a-4398-ae0f-ae6205d6134a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:14.026419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0dc1880d-240a-4398-ae0f-ae6205d6134a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:14.034771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 688/1370] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:14.079051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc333c7-97e3-4253-be32-d7557a60a36e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:14.095386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc333c7-97e3-4253-be32-d7557a60a36e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:14.101662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcf98a2-01fa-4a45-8916-98275342039a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:14.102739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcf98a2-01fa-4a45-8916-98275342039a service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:14.103203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcf98a2-01fa-4a45-8916-98275342039a service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:14.103662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcf98a2-01fa-4a45-8916-98275342039a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:14.113395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcf98a2-01fa-4a45-8916-98275342039a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:14.115333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcf98a2-01fa-4a45-8916-98275342039a service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:14.125669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 683/1371] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:14.753472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c27fb-ab12-470c-a608-293b8c94ef8b None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:14.753762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c27fb-ab12-470c-a608-293b8c94ef8b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:14.754305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c27fb-ab12-470c-a608-293b8c94ef8b None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:14.755768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 689/1372] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:14.761565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f41a3a-3cce-4e60-b948-dc5bdd1729cd None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:14.761860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f41a3a-3cce-4e60-b948-dc5bdd1729cd None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:14.762146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f41a3a-3cce-4e60-b948-dc5bdd1729cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:14.781219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f2f41a3a-3cce-4e60-b948-dc5bdd1729cd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:14.782553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f2f41a3a-3cce-4e60-b948-dc5bdd1729cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:14.791538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 684/1373] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:15.129330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f945f1c-22dd-40ba-872d-4533eae66fda None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:15.129330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f945f1c-22dd-40ba-872d-4533eae66fda None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:15.129330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f945f1c-22dd-40ba-872d-4533eae66fda None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:15.130457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 690/1374] 10.222.0.20 () {54 vars in 963 bytes} [Tue Jul 5 10:03:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:15.136733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393b0137-c410-499b-916f-ea6ce9214839 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:15.137151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393b0137-c410-499b-916f-ea6ce9214839 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:15.137316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393b0137-c410-499b-916f-ea6ce9214839 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:15.188898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-393b0137-c410-499b-916f-ea6ce9214839 None None] MFA Rules not processed for user `1fc5b3da67de48cb83848168b16dd153`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:15.190093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-393b0137-c410-499b-916f-ea6ce9214839 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:15.255448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 685/1375] 10.222.0.20 () {58 vars in 1060 bytes} [Tue Jul 5 10:03:15 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:15.262638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f945f1c-22dd-40ba-872d-4533eae66fda None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:15.275271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f945f1c-22dd-40ba-872d-4533eae66fda None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:15.279305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d42811-a40c-4195-bdab-d269890578e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:15.280239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d42811-a40c-4195-bdab-d269890578e6 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:15.280774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d42811-a40c-4195-bdab-d269890578e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:15.281277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d42811-a40c-4195-bdab-d269890578e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:15.291909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d42811-a40c-4195-bdab-d269890578e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:15.294028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d42811-a40c-4195-bdab-d269890578e6 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:15.305957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 691/1376] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:16.187821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6e4756-887c-4d16-8584-98fc419b80a5 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:16.188971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6e4756-887c-4d16-8584-98fc419b80a5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:16.189463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6e4756-887c-4d16-8584-98fc419b80a5 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:16.191067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 686/1377] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:16.196501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86d4529-9386-4bf3-bba5-128e6513afb8 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:16.198164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86d4529-9386-4bf3-bba5-128e6513afb8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:16.198639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86d4529-9386-4bf3-bba5-128e6513afb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:16.217912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b86d4529-9386-4bf3-bba5-128e6513afb8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:16.219130 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b86d4529-9386-4bf3-bba5-128e6513afb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:16.227603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 692/1378] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:16.270041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6e4756-887c-4d16-8584-98fc419b80a5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:16.284913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6e4756-887c-4d16-8584-98fc419b80a5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:16.291130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17513e39-8f84-4d44-aef2-5e60570a58a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:16.291767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17513e39-8f84-4d44-aef2-5e60570a58a2 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:16.292050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17513e39-8f84-4d44-aef2-5e60570a58a2 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:16.292329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17513e39-8f84-4d44-aef2-5e60570a58a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:16.300457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17513e39-8f84-4d44-aef2-5e60570a58a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:16.301992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17513e39-8f84-4d44-aef2-5e60570a58a2 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:16.305892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:03:16.312423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 687/1379] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:18.386149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465b9c32-128d-4c9c-a444-cfabe61a245f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:18.386644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465b9c32-128d-4c9c-a444-cfabe61a245f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:18.386738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465b9c32-128d-4c9c-a444-cfabe61a245f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:18.388102 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 693/1380] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:18.393359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88215562-968f-47fe-8c6a-b99c670eba32 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:18.393786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88215562-968f-47fe-8c6a-b99c670eba32 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:18.393925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88215562-968f-47fe-8c6a-b99c670eba32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:18.412281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-88215562-968f-47fe-8c6a-b99c670eba32 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:18.414422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-88215562-968f-47fe-8c6a-b99c670eba32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:18.422110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 688/1381] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:18.465243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465b9c32-128d-4c9c-a444-cfabe61a245f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:18.480028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465b9c32-128d-4c9c-a444-cfabe61a245f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:18.485672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c9a5d2-ae71-4ea9-a638-d868cf6d632b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:18.486304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c9a5d2-ae71-4ea9-a638-d868cf6d632b service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:18.486589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c9a5d2-ae71-4ea9-a638-d868cf6d632b service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:18.486868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c9a5d2-ae71-4ea9-a638-d868cf6d632b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:18.494943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c9a5d2-ae71-4ea9-a638-d868cf6d632b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:18.496564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c9a5d2-ae71-4ea9-a638-d868cf6d632b service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:18.508757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 694/1382] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:20.552705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d7ce2-6eb2-4b06-80e8-cd0a44f54c5e None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:20.553241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d7ce2-6eb2-4b06-80e8-cd0a44f54c5e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:20.553241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84d7ce2-6eb2-4b06-80e8-cd0a44f54c5e None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:20.554656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 689/1383] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:20.559742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68819a1-e460-4d2c-ad69-ea2c63abff15 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:20.560135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68819a1-e460-4d2c-ad69-ea2c63abff15 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:20.560289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68819a1-e460-4d2c-ad69-ea2c63abff15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:20.578384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f68819a1-e460-4d2c-ad69-ea2c63abff15 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:20.579727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f68819a1-e460-4d2c-ad69-ea2c63abff15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:20.588263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 695/1384] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:20.630062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84d7ce2-6eb2-4b06-80e8-cd0a44f54c5e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:20.644570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84d7ce2-6eb2-4b06-80e8-cd0a44f54c5e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:20.650426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8895b47d-1fef-4a18-b7ee-a2a481b3b831 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:20.651069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8895b47d-1fef-4a18-b7ee-a2a481b3b831 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:20.651352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8895b47d-1fef-4a18-b7ee-a2a481b3b831 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:20.651628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8895b47d-1fef-4a18-b7ee-a2a481b3b831 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:20.660586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8895b47d-1fef-4a18-b7ee-a2a481b3b831 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:20.662406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8895b47d-1fef-4a18-b7ee-a2a481b3b831 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:20.673553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 690/1385] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:22.737820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a80c78-3dba-4b84-a0d5-e17b947d5a19 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:22.738293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a80c78-3dba-4b84-a0d5-e17b947d5a19 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:22.738946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a80c78-3dba-4b84-a0d5-e17b947d5a19 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:22.742087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 696/1386] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:22.746469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1af2cb-d5b1-441d-b547-9a0d0003c3fa None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:22.746890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1af2cb-d5b1-441d-b547-9a0d0003c3fa None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:22.747242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1af2cb-d5b1-441d-b547-9a0d0003c3fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:22.777313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2b1af2cb-d5b1-441d-b547-9a0d0003c3fa None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:22.780362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2b1af2cb-d5b1-441d-b547-9a0d0003c3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:22.792690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 691/1387] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:22.835624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a80c78-3dba-4b84-a0d5-e17b947d5a19 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:22.850597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a80c78-3dba-4b84-a0d5-e17b947d5a19 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:22.856205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bd3c2d-6f2e-4cc8-8db5-a99b732025a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:22.856951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bd3c2d-6f2e-4cc8-8db5-a99b732025a9 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:22.857237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bd3c2d-6f2e-4cc8-8db5-a99b732025a9 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:22.857514 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bd3c2d-6f2e-4cc8-8db5-a99b732025a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:22.867001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bd3c2d-6f2e-4cc8-8db5-a99b732025a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:22.868608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bd3c2d-6f2e-4cc8-8db5-a99b732025a9 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:22.878440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 697/1388] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:25.101599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475106eb-cab0-4c26-abdb-c9f46361ceaa None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:25.102140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475106eb-cab0-4c26-abdb-c9f46361ceaa None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:25.102140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475106eb-cab0-4c26-abdb-c9f46361ceaa None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:25.103603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 692/1389] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:25.109271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90cf88f-de23-433b-844e-4e52080548d5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:25.109702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90cf88f-de23-433b-844e-4e52080548d5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:25.110035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90cf88f-de23-433b-844e-4e52080548d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:25.133166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c90cf88f-de23-433b-844e-4e52080548d5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:25.134759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c90cf88f-de23-433b-844e-4e52080548d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:25.145041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 698/1390] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:25.190094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475106eb-cab0-4c26-abdb-c9f46361ceaa None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:25.205393 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475106eb-cab0-4c26-abdb-c9f46361ceaa None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:25.210857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123bf49c-2250-40a2-9835-b19fdfa94197 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:25.211511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123bf49c-2250-40a2-9835-b19fdfa94197 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:25.211797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123bf49c-2250-40a2-9835-b19fdfa94197 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:25.212076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123bf49c-2250-40a2-9835-b19fdfa94197 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:25.220679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123bf49c-2250-40a2-9835-b19fdfa94197 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:25.222272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123bf49c-2250-40a2-9835-b19fdfa94197 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:25.231949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 693/1391] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:27.302631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc18c04-794d-4181-b671-e858222a08b7 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:27.303190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc18c04-794d-4181-b671-e858222a08b7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:27.303666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc18c04-794d-4181-b671-e858222a08b7 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:27.307315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 699/1392] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:27 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:27.312593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba07032-8bc6-462d-b003-b6883538efe7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:27.313024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba07032-8bc6-462d-b003-b6883538efe7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:27.313154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba07032-8bc6-462d-b003-b6883538efe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:27.338966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-bba07032-8bc6-462d-b003-b6883538efe7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:27.340188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bba07032-8bc6-462d-b003-b6883538efe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:27.348190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 694/1393] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:27.393209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc18c04-794d-4181-b671-e858222a08b7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:27.419301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc18c04-794d-4181-b671-e858222a08b7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:27.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712c4aef-a052-46c5-ac37-0fb3e2c3bb95 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:27.428138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712c4aef-a052-46c5-ac37-0fb3e2c3bb95 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:27.428521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712c4aef-a052-46c5-ac37-0fb3e2c3bb95 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:27.428736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712c4aef-a052-46c5-ac37-0fb3e2c3bb95 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:27.439943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712c4aef-a052-46c5-ac37-0fb3e2c3bb95 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:27.442926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712c4aef-a052-46c5-ac37-0fb3e2c3bb95 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:27.461370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 700/1394] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:29.499952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fe2943-5abb-4400-bdc2-eeef9765d5e4 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:29.500536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fe2943-5abb-4400-bdc2-eeef9765d5e4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:29.500536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fe2943-5abb-4400-bdc2-eeef9765d5e4 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:29.501923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 695/1395] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:29.506997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340f3291-2bd1-47ce-9a96-53b34cdc5899 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:29.507506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340f3291-2bd1-47ce-9a96-53b34cdc5899 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:29.507632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340f3291-2bd1-47ce-9a96-53b34cdc5899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:29.526387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-340f3291-2bd1-47ce-9a96-53b34cdc5899 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:29.527587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-340f3291-2bd1-47ce-9a96-53b34cdc5899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:29.535430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 701/1396] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:29.578655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fe2943-5abb-4400-bdc2-eeef9765d5e4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:29.593407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fe2943-5abb-4400-bdc2-eeef9765d5e4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:29.598701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712e3bb6-e6bb-43a8-b167-8fc773349d34 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:29.599338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712e3bb6-e6bb-43a8-b167-8fc773349d34 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:29.599619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712e3bb6-e6bb-43a8-b167-8fc773349d34 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:29.599892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712e3bb6-e6bb-43a8-b167-8fc773349d34 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:29.608475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712e3bb6-e6bb-43a8-b167-8fc773349d34 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:29.610054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712e3bb6-e6bb-43a8-b167-8fc773349d34 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:29.619268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 696/1397] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:30.390910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7f3436-6fb4-4a4d-ae13-4076f95c0e9b None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:30.391201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7f3436-6fb4-4a4d-ae13-4076f95c0e9b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:30.391479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7f3436-6fb4-4a4d-ae13-4076f95c0e9b None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:30.392929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 702/1398] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:30.398278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb52ad3a-4360-42af-b02b-d9b37f1a1182 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:30.398570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb52ad3a-4360-42af-b02b-d9b37f1a1182 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:30.398853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb52ad3a-4360-42af-b02b-d9b37f1a1182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:30.416686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-fb52ad3a-4360-42af-b02b-d9b37f1a1182 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:30.417885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fb52ad3a-4360-42af-b02b-d9b37f1a1182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:30.425653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 697/1399] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:30.480552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3cd18-e5ca-4232-adf5-b94340262558 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:30.480847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3cd18-e5ca-4232-adf5-b94340262558 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:30.481124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb3cd18-e5ca-4232-adf5-b94340262558 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:30.482500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 703/1400] 10.222.0.20 () {54 vars in 962 bytes} [Tue Jul 5 10:03:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:30.487427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f794025-f88f-4175-b11d-597f458c47cd None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:30.487718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f794025-f88f-4175-b11d-597f458c47cd None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:30.487994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f794025-f88f-4175-b11d-597f458c47cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:30.536677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8f794025-f88f-4175-b11d-597f458c47cd None None] MFA Rules not processed for user `90d9b9421d454465b8f99137cd182abf`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:30.537879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8f794025-f88f-4175-b11d-597f458c47cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:30.597052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 698/1401] 10.222.0.20 () {58 vars in 1059 bytes} [Tue Jul 5 10:03:30 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:30.605131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3cd18-e5ca-4232-adf5-b94340262558 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:30.615642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb3cd18-e5ca-4232-adf5-b94340262558 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:30.619762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad34916-47c0-4227-8f6e-2eb88fc2bf30 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:30.620414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad34916-47c0-4227-8f6e-2eb88fc2bf30 service glance] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:30.620708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad34916-47c0-4227-8f6e-2eb88fc2bf30 service glance] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:30.620993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad34916-47c0-4227-8f6e-2eb88fc2bf30 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:30.629951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad34916-47c0-4227-8f6e-2eb88fc2bf30 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:30.631318 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad34916-47c0-4227-8f6e-2eb88fc2bf30 service glance] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:30.640763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 704/1402] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:31.347544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b0f33-0b75-4bb6-ac05-ff482203578c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:31.347919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b0f33-0b75-4bb6-ac05-ff482203578c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:31.348294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0b0f33-0b75-4bb6-ac05-ff482203578c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:31.395335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4d0b0f33-0b75-4bb6-ac05-ff482203578c None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:31.396538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4d0b0f33-0b75-4bb6-ac05-ff482203578c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:31.435891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:03:31.461151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 699/1403] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:03:31 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:31.504531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c476221e-cf82-47f4-82be-d9f29867800f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:31.504829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c476221e-cf82-47f4-82be-d9f29867800f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:31.505112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c476221e-cf82-47f4-82be-d9f29867800f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:31.506511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 705/1404] 10.222.0.20 () {54 vars in 942 bytes} [Tue Jul 5 10:03:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:31.513495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1072647-1207-4cb7-8fc2-16362466e26e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:31.513853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1072647-1207-4cb7-8fc2-16362466e26e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:31.514209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1072647-1207-4cb7-8fc2-16362466e26e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:31.562874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a1072647-1207-4cb7-8fc2-16362466e26e None None] MFA Rules not processed for user `c1d16fdc498e4d84bf8657c7f55d6b26`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:31.563997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a1072647-1207-4cb7-8fc2-16362466e26e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:31.636247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 700/1405] 10.222.0.20 () {58 vars in 1039 bytes} [Tue Jul 5 10:03:31 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:31.645717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c476221e-cf82-47f4-82be-d9f29867800f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:31.652788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a37cc0-b1ac-40d9-b28f-40716d8d865d None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:31.653073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a37cc0-b1ac-40d9-b28f-40716d8d865d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:31.653351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a37cc0-b1ac-40d9-b28f-40716d8d865d None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:31.654706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 701/1406] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:03:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 10:03:31.659516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0acf7-b103-42c5-b204-e0a1002649d0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:31.659794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0acf7-b103-42c5-b204-e0a1002649d0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:31.659922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c476221e-cf82-47f4-82be-d9f29867800f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:31.660065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0acf7-b103-42c5-b204-e0a1002649d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:31.665064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:31.665707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:31.665995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:31.666452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:31.676012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:31.677670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:31.677847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-39a0acf7-b103-42c5-b204-e0a1002649d0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:31.679121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-39a0acf7-b103-42c5-b204-e0a1002649d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:31.687704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 706/1407] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:03:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:03:31.688011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 702/1408] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:03:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:03:31.731877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:31.744294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5beccb0a-ce39-4f0a-bc8a-1047cb848464 service swift] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:31.750451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e117304e-b8dc-4fcb-85d2-f8c37052f31e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:31.751245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e117304e-b8dc-4fcb-85d2-f8c37052f31e service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:31.751533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e117304e-b8dc-4fcb-85d2-f8c37052f31e service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:31.751813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e117304e-b8dc-4fcb-85d2-f8c37052f31e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:31.761581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e117304e-b8dc-4fcb-85d2-f8c37052f31e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:31.762869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e117304e-b8dc-4fcb-85d2-f8c37052f31e service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:31.773005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 707/1409] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:41.672585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=56 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:03:41.768494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=72 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:03:53.412630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b08d6d-a92b-499c-a578-c81929b72acc None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:53.413217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b08d6d-a92b-499c-a578-c81929b72acc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:53.413217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b08d6d-a92b-499c-a578-c81929b72acc None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:53.414550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 703/1410] 10.222.0.20 () {54 vars in 918 bytes} [Tue Jul 5 10:03:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:53.419828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1edb25-72f0-44ba-b44c-0c3f9fddc77d None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:53.420266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1edb25-72f0-44ba-b44c-0c3f9fddc77d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:53.420371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1edb25-72f0-44ba-b44c-0c3f9fddc77d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:53.438425 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ba1edb25-72f0-44ba-b44c-0c3f9fddc77d None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:53.439629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ba1edb25-72f0-44ba-b44c-0c3f9fddc77d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:53.448638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 708/1411] 10.222.0.20 () {58 vars in 1015 bytes} [Tue Jul 5 10:03:53 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:53.462305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6936baf5-d0b6-4b4b-8421-93c90ee4e938 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:53.462592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6936baf5-d0b6-4b4b-8421-93c90ee4e938 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:53.462870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6936baf5-d0b6-4b4b-8421-93c90ee4e938 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:53.464192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 704/1412] 10.222.0.20 () {54 vars in 918 bytes} [Tue Jul 5 10:03:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:53.468892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a07df41-3da2-4e14-92e8-0037f06b70f3 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:53.469177 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a07df41-3da2-4e14-92e8-0037f06b70f3 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:53.469453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a07df41-3da2-4e14-92e8-0037f06b70f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:53.486767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9a07df41-3da2-4e14-92e8-0037f06b70f3 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:53.487948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9a07df41-3da2-4e14-92e8-0037f06b70f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:53.497532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 709/1413] 10.222.0.20 () {58 vars in 1015 bytes} [Tue Jul 5 10:03:53 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:03:55.935545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6d14c-1aeb-4e1f-ac75-997fcda837aa None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:55.936036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6d14c-1aeb-4e1f-ac75-997fcda837aa None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:55.936143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b6d14c-1aeb-4e1f-ac75-997fcda837aa None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:55.937661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 705/1414] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:03:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:55.942856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca4aebb-15b1-4f3a-ac15-33a95561c203 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:55.943283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca4aebb-15b1-4f3a-ac15-33a95561c203 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:55.943446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca4aebb-15b1-4f3a-ac15-33a95561c203 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:55.961408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-aca4aebb-15b1-4f3a-ac15-33a95561c203 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:55.963126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-aca4aebb-15b1-4f3a-ac15-33a95561c203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:55.970810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 710/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:03:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:56.013915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b6d14c-1aeb-4e1f-ac75-997fcda837aa None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:56.028846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b6d14c-1aeb-4e1f-ac75-997fcda837aa None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:56.034157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fdd9f7-585d-425d-9c20-0ae5969a90cf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:56.034785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdd9f7-585d-425d-9c20-0ae5969a90cf service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:56.035063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdd9f7-585d-425d-9c20-0ae5969a90cf service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:56.035335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdd9f7-585d-425d-9c20-0ae5969a90cf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:56.044073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fdd9f7-585d-425d-9c20-0ae5969a90cf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:56.045688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fdd9f7-585d-425d-9c20-0ae5969a90cf service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:56.055156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 706/1416] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:56.941652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24342a2-9c4d-4667-bed8-eba3ffeabf05 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:56.941948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24342a2-9c4d-4667-bed8-eba3ffeabf05 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:56.942233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24342a2-9c4d-4667-bed8-eba3ffeabf05 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:56.943637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 711/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:03:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:56.948474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2cc319-e648-4f90-8bd8-ffc41c640eed None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:56.948849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2cc319-e648-4f90-8bd8-ffc41c640eed None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:56.949031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2cc319-e648-4f90-8bd8-ffc41c640eed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:56.966456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ca2cc319-e648-4f90-8bd8-ffc41c640eed None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:56.967662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ca2cc319-e648-4f90-8bd8-ffc41c640eed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:56.975341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 707/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:03:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:57.123486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24342a2-9c4d-4667-bed8-eba3ffeabf05 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:57.138558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24342a2-9c4d-4667-bed8-eba3ffeabf05 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:57.143992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a735a931-e121-4185-b46f-3a2f4d705e93 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:57.144671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a735a931-e121-4185-b46f-3a2f4d705e93 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:57.144949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a735a931-e121-4185-b46f-3a2f4d705e93 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:57.145230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a735a931-e121-4185-b46f-3a2f4d705e93 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:57.154148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a735a931-e121-4185-b46f-3a2f4d705e93 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:57.155725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a735a931-e121-4185-b46f-3a2f4d705e93 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:57.165610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 712/1419] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:58.028823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a01e406-f4eb-4c80-bbba-8179f6efd4d2 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:58.029353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a01e406-f4eb-4c80-bbba-8179f6efd4d2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:58.029353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a01e406-f4eb-4c80-bbba-8179f6efd4d2 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:58.030777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 708/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:03:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:58.035552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778dd32e-5d99-45e3-b9e9-0293d4939acd None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:58.035974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778dd32e-5d99-45e3-b9e9-0293d4939acd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:58.036108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778dd32e-5d99-45e3-b9e9-0293d4939acd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:58.054660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-778dd32e-5d99-45e3-b9e9-0293d4939acd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:58.055933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-778dd32e-5d99-45e3-b9e9-0293d4939acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:58.063584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 713/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:03:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:58.212782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a01e406-f4eb-4c80-bbba-8179f6efd4d2 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:58.227026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a01e406-f4eb-4c80-bbba-8179f6efd4d2 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:58.233125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100d2d52-7d5c-46ed-8c6b-0a7db15a4d81 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:58.233762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100d2d52-7d5c-46ed-8c6b-0a7db15a4d81 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:58.234044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100d2d52-7d5c-46ed-8c6b-0a7db15a4d81 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:58.234319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100d2d52-7d5c-46ed-8c6b-0a7db15a4d81 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:58.242468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100d2d52-7d5c-46ed-8c6b-0a7db15a4d81 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:58.244059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100d2d52-7d5c-46ed-8c6b-0a7db15a4d81 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:58.253296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 709/1422] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:03:59.115813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c08fe6-42be-45b5-a8c4-489639ca2da7 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:59.116360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c08fe6-42be-45b5-a8c4-489639ca2da7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:59.116360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c08fe6-42be-45b5-a8c4-489639ca2da7 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:59.117817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 714/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:03:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:03:59.123114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc6834-4f09-4304-affa-dbc6f348f387 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:59.123509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc6834-4f09-4304-affa-dbc6f348f387 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:59.123670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffc6834-4f09-4304-affa-dbc6f348f387 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:59.141403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6ffc6834-4f09-4304-affa-dbc6f348f387 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:03:59.142597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6ffc6834-4f09-4304-affa-dbc6f348f387 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:03:59.150268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 710/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:03:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:03:59.298677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c08fe6-42be-45b5-a8c4-489639ca2da7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:03:59.313040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c08fe6-42be-45b5-a8c4-489639ca2da7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:03:59.318414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129d3416-3f7b-46c8-9f38-fa57c4bb814f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:03:59.319057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d3416-3f7b-46c8-9f38-fa57c4bb814f service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:03:59.319338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d3416-3f7b-46c8-9f38-fa57c4bb814f service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:03:59.319638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d3416-3f7b-46c8-9f38-fa57c4bb814f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:03:59.327709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129d3416-3f7b-46c8-9f38-fa57c4bb814f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:03:59.329309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129d3416-3f7b-46c8-9f38-fa57c4bb814f service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:03:59.338466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 715/1425] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:03:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:00.188803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b2f40-f559-4cb8-bd98-3f22a0265f51 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:00.189345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b2f40-f559-4cb8-bd98-3f22a0265f51 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:00.189510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b2f40-f559-4cb8-bd98-3f22a0265f51 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:00.190884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 711/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:04:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:00.195742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a610c-d794-414e-b35c-efab312be67a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:00.196166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a610c-d794-414e-b35c-efab312be67a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:00.196924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a610c-d794-414e-b35c-efab312be67a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:00.214576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a31a610c-d794-414e-b35c-efab312be67a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:00.215785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a31a610c-d794-414e-b35c-efab312be67a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:00.223621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 716/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:04:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:00.372336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b2f40-f559-4cb8-bd98-3f22a0265f51 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:00.387087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b2f40-f559-4cb8-bd98-3f22a0265f51 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:00.392776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186b10cb-ef00-4e46-87ed-65817f334c90 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:00.393458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186b10cb-ef00-4e46-87ed-65817f334c90 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:00.393743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186b10cb-ef00-4e46-87ed-65817f334c90 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:00.394025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186b10cb-ef00-4e46-87ed-65817f334c90 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:00.402437 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186b10cb-ef00-4e46-87ed-65817f334c90 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:00.404023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186b10cb-ef00-4e46-87ed-65817f334c90 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:00.413400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 712/1428] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:01.209185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb4740e-1b42-4995-8274-fe27d2e34021 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:01.209705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb4740e-1b42-4995-8274-fe27d2e34021 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:01.209822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb4740e-1b42-4995-8274-fe27d2e34021 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:01.211214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 717/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:04:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:01.216543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e5268-dfba-4a21-ac9d-5ab377878d8c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:01.216966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e5268-dfba-4a21-ac9d-5ab377878d8c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:01.217104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e5268-dfba-4a21-ac9d-5ab377878d8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:01.235402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-554e5268-dfba-4a21-ac9d-5ab377878d8c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:01.236636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-554e5268-dfba-4a21-ac9d-5ab377878d8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:01.244291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 713/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:04:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:01.394594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb4740e-1b42-4995-8274-fe27d2e34021 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:01.408922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb4740e-1b42-4995-8274-fe27d2e34021 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:01.414244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b399e63-af8b-448c-88b5-82cbfcb9b694 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:01.414880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b399e63-af8b-448c-88b5-82cbfcb9b694 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:01.415664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b399e63-af8b-448c-88b5-82cbfcb9b694 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:01.415942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b399e63-af8b-448c-88b5-82cbfcb9b694 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:01.424260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b399e63-af8b-448c-88b5-82cbfcb9b694 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:01.425875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b399e63-af8b-448c-88b5-82cbfcb9b694 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:01.435824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 718/1431] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:02.225850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7df5c3-e983-4dcf-a841-6dcf02b1ddbe None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:02.226417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7df5c3-e983-4dcf-a841-6dcf02b1ddbe None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:02.226417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7df5c3-e983-4dcf-a841-6dcf02b1ddbe None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:02.227795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 714/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:04:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:02.232867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301355cf-23f9-41b0-8342-457095c92132 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:02.233292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301355cf-23f9-41b0-8342-457095c92132 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:02.233404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301355cf-23f9-41b0-8342-457095c92132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:02.250749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-301355cf-23f9-41b0-8342-457095c92132 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:02.251931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-301355cf-23f9-41b0-8342-457095c92132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:02.260148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 719/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:04:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:02.407402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7df5c3-e983-4dcf-a841-6dcf02b1ddbe None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:02.421590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7df5c3-e983-4dcf-a841-6dcf02b1ddbe None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:02.426826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a61c8f-a8d1-4af7-846a-c09732482f63 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:02.427462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a61c8f-a8d1-4af7-846a-c09732482f63 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:02.427742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a61c8f-a8d1-4af7-846a-c09732482f63 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:02.428016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a61c8f-a8d1-4af7-846a-c09732482f63 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:02.436380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a61c8f-a8d1-4af7-846a-c09732482f63 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:02.437983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a61c8f-a8d1-4af7-846a-c09732482f63 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:02.447953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 715/1434] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:03.238385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43dc20-223d-4f92-9428-73f819695849 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:03.238918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43dc20-223d-4f92-9428-73f819695849 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:03.239043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43dc20-223d-4f92-9428-73f819695849 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:03.240329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 720/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:04:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:03.245392 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47965d-38b5-43c4-bf95-b8cddcf06244 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:03.245767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47965d-38b5-43c4-bf95-b8cddcf06244 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:03.245960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f47965d-38b5-43c4-bf95-b8cddcf06244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:03.263479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5f47965d-38b5-43c4-bf95-b8cddcf06244 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:03.264720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5f47965d-38b5-43c4-bf95-b8cddcf06244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:03.272511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 716/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:04:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:03.422616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b43dc20-223d-4f92-9428-73f819695849 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:03.436829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b43dc20-223d-4f92-9428-73f819695849 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:03.442113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c4c31f-b7c4-4772-ba64-1c93021f096d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:03.443022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c4c31f-b7c4-4772-ba64-1c93021f096d service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:03.443314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c4c31f-b7c4-4772-ba64-1c93021f096d service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:03.443594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c4c31f-b7c4-4772-ba64-1c93021f096d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:03.451516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c4c31f-b7c4-4772-ba64-1c93021f096d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:03.453123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c4c31f-b7c4-4772-ba64-1c93021f096d service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:03.463167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 721/1437] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:04.276040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7836bc02-17d7-44a2-ad9b-8ba2878637cf None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:04.276603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7836bc02-17d7-44a2-ad9b-8ba2878637cf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:04.276731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7836bc02-17d7-44a2-ad9b-8ba2878637cf None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:04.278091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 717/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:04:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:04.282858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e90758d-1b20-42d4-a88a-ae2043e02126 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:04.283239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e90758d-1b20-42d4-a88a-ae2043e02126 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:04.283417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e90758d-1b20-42d4-a88a-ae2043e02126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:04.301202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0e90758d-1b20-42d4-a88a-ae2043e02126 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:04.302390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0e90758d-1b20-42d4-a88a-ae2043e02126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:04.310625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 722/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:04:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:04.469552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7836bc02-17d7-44a2-ad9b-8ba2878637cf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:04.484170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7836bc02-17d7-44a2-ad9b-8ba2878637cf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:04.489554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c6eb3c-1452-46c3-bcf1-d12fcd9e9384 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:04.490187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c6eb3c-1452-46c3-bcf1-d12fcd9e9384 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:04.490471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c6eb3c-1452-46c3-bcf1-d12fcd9e9384 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:04.490756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c6eb3c-1452-46c3-bcf1-d12fcd9e9384 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:04.499354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c6eb3c-1452-46c3-bcf1-d12fcd9e9384 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:04.500975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c6eb3c-1452-46c3-bcf1-d12fcd9e9384 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:04.510166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 718/1440] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:06.431795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7769b-50ae-4b08-9ec0-f839e4fc4af8 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:06.432308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7769b-50ae-4b08-9ec0-f839e4fc4af8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:06.432435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7769b-50ae-4b08-9ec0-f839e4fc4af8 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:06.433778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 723/1441] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:06.439074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda509e-dfe1-4006-952e-21e65bb588b9 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:06.439496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda509e-dfe1-4006-952e-21e65bb588b9 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:06.439634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fda509e-dfe1-4006-952e-21e65bb588b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:06.456878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8fda509e-dfe1-4006-952e-21e65bb588b9 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:06.458059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8fda509e-dfe1-4006-952e-21e65bb588b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:06.466390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 719/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:06.507280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a7769b-50ae-4b08-9ec0-f839e4fc4af8 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:06.519153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a7769b-50ae-4b08-9ec0-f839e4fc4af8 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:06.522610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f451b931-0aa6-41e7-a407-3926e6d8ba33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:06.523245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f451b931-0aa6-41e7-a407-3926e6d8ba33 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:06.523529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f451b931-0aa6-41e7-a407-3926e6d8ba33 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:06.523803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f451b931-0aa6-41e7-a407-3926e6d8ba33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:06.533142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f451b931-0aa6-41e7-a407-3926e6d8ba33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:06.534570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f451b931-0aa6-41e7-a407-3926e6d8ba33 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:06.544584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 724/1443] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:08.693832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fad0d3-c2ff-4652-9bf5-2e195ab48e45 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:08.694516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fad0d3-c2ff-4652-9bf5-2e195ab48e45 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:08.694516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fad0d3-c2ff-4652-9bf5-2e195ab48e45 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:08.696087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 720/1444] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:08.701378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487a5f6-f0b4-4fd7-a437-b6d36cc9fc3e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:08.701791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487a5f6-f0b4-4fd7-a437-b6d36cc9fc3e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:08.701937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487a5f6-f0b4-4fd7-a437-b6d36cc9fc3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:08.719732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e487a5f6-f0b4-4fd7-a437-b6d36cc9fc3e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:08.720952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e487a5f6-f0b4-4fd7-a437-b6d36cc9fc3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:08.729213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 725/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:08.765882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fad0d3-c2ff-4652-9bf5-2e195ab48e45 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:08.777159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fad0d3-c2ff-4652-9bf5-2e195ab48e45 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:08.780598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf5cbf8-a0f1-4f85-a346-1316ade13b3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:08.781226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5cbf8-a0f1-4f85-a346-1316ade13b3d service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:08.781506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5cbf8-a0f1-4f85-a346-1316ade13b3d service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:08.781779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf5cbf8-a0f1-4f85-a346-1316ade13b3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:08.791093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf5cbf8-a0f1-4f85-a346-1316ade13b3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:08.792812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf5cbf8-a0f1-4f85-a346-1316ade13b3d service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:08.801878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 721/1446] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:11.106637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceae15e-77e2-428f-b851-74e8e9e5acc2 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:11.107201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceae15e-77e2-428f-b851-74e8e9e5acc2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:11.107201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceae15e-77e2-428f-b851-74e8e9e5acc2 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:11.108750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 726/1447] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:11.113834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cec879-7495-4ce4-9cc4-26cdd11d83d7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:11.114277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cec879-7495-4ce4-9cc4-26cdd11d83d7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:11.114384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cec879-7495-4ce4-9cc4-26cdd11d83d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:11.132980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-72cec879-7495-4ce4-9cc4-26cdd11d83d7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:11.134241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-72cec879-7495-4ce4-9cc4-26cdd11d83d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:11.142080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 722/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:11.180045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceae15e-77e2-428f-b851-74e8e9e5acc2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:11.191533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceae15e-77e2-428f-b851-74e8e9e5acc2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:11.195033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fa1478-80eb-456a-911e-a35b496eebca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:11.195668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fa1478-80eb-456a-911e-a35b496eebca service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:11.195948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fa1478-80eb-456a-911e-a35b496eebca service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:11.196220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fa1478-80eb-456a-911e-a35b496eebca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:11.205366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fa1478-80eb-456a-911e-a35b496eebca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:11.206796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fa1478-80eb-456a-911e-a35b496eebca service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:11.216238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 727/1449] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:13.343207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17065c-186c-451c-9406-58e85f693941 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:13.343760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17065c-186c-451c-9406-58e85f693941 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:13.343867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17065c-186c-451c-9406-58e85f693941 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:13.345208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 723/1450] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:13.350451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994fb323-5c69-4b2d-87cb-ebb3b19a80b7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:13.350879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994fb323-5c69-4b2d-87cb-ebb3b19a80b7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:13.351005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994fb323-5c69-4b2d-87cb-ebb3b19a80b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:13.368116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-994fb323-5c69-4b2d-87cb-ebb3b19a80b7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:13.369347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-994fb323-5c69-4b2d-87cb-ebb3b19a80b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:13.377196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 728/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:13.414950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce17065c-186c-451c-9406-58e85f693941 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:13.425718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce17065c-186c-451c-9406-58e85f693941 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:13.429162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43735fd-0c45-4d48-a3a3-83ba79f6356b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:13.429792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43735fd-0c45-4d48-a3a3-83ba79f6356b service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:13.430072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43735fd-0c45-4d48-a3a3-83ba79f6356b service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:13.430343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43735fd-0c45-4d48-a3a3-83ba79f6356b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:13.438930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43735fd-0c45-4d48-a3a3-83ba79f6356b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:13.440343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43735fd-0c45-4d48-a3a3-83ba79f6356b service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:13.449989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 724/1452] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:15.617419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9f3830-ac9d-4b6b-b58e-6250e1ee843b None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:15.617978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9f3830-ac9d-4b6b-b58e-6250e1ee843b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:15.617978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9f3830-ac9d-4b6b-b58e-6250e1ee843b None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:15.619945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 729/1453] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:15.625430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744884d0-b65e-4f7a-9014-794adf451c87 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:15.625855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744884d0-b65e-4f7a-9014-794adf451c87 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:15.625987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744884d0-b65e-4f7a-9014-794adf451c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:15.643342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-744884d0-b65e-4f7a-9014-794adf451c87 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:15.644579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-744884d0-b65e-4f7a-9014-794adf451c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:15.652242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 725/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:15.690910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9f3830-ac9d-4b6b-b58e-6250e1ee843b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:15.701393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9f3830-ac9d-4b6b-b58e-6250e1ee843b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:15.704827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9da047-2bef-44ea-91da-b0dd50e16f36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:15.705454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9da047-2bef-44ea-91da-b0dd50e16f36 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:15.705735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9da047-2bef-44ea-91da-b0dd50e16f36 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:15.706011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9da047-2bef-44ea-91da-b0dd50e16f36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:15.714614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9da047-2bef-44ea-91da-b0dd50e16f36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:15.716030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9da047-2bef-44ea-91da-b0dd50e16f36 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:15.725164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 730/1455] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:17.793545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75e397-8473-4015-b026-bc0e6ef7bfb3 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:17.794115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75e397-8473-4015-b026-bc0e6ef7bfb3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:17.794115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd75e397-8473-4015-b026-bc0e6ef7bfb3 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:17.795471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 726/1456] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:17.800864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e147b-ded1-4fe1-bf1f-66e4971caeea None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:17.801251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e147b-ded1-4fe1-bf1f-66e4971caeea None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:17.801412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e147b-ded1-4fe1-bf1f-66e4971caeea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:17.819378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-598e147b-ded1-4fe1-bf1f-66e4971caeea None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:17.820621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-598e147b-ded1-4fe1-bf1f-66e4971caeea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:17.829098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 731/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:17.865860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75e397-8473-4015-b026-bc0e6ef7bfb3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:17.877072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd75e397-8473-4015-b026-bc0e6ef7bfb3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:17.880585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcb6d11-aace-4cc3-9eed-5eaa863102d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:17.881215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb6d11-aace-4cc3-9eed-5eaa863102d1 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:17.881499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb6d11-aace-4cc3-9eed-5eaa863102d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:17.881774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb6d11-aace-4cc3-9eed-5eaa863102d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:17.890530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcb6d11-aace-4cc3-9eed-5eaa863102d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:17.891957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcb6d11-aace-4cc3-9eed-5eaa863102d1 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:17.901474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 727/1458] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:19.976162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf84cba-a309-4a6c-be65-272a7ae7b928 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:19.976668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf84cba-a309-4a6c-be65-272a7ae7b928 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:19.976760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf84cba-a309-4a6c-be65-272a7ae7b928 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:19.978137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 732/1459] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:19.983896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f1976-225c-4ed2-9475-7c34fb1af38b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:19.984318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f1976-225c-4ed2-9475-7c34fb1af38b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:19.984513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f1976-225c-4ed2-9475-7c34fb1af38b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:20.001829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0c8f1976-225c-4ed2-9475-7c34fb1af38b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:20.003016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0c8f1976-225c-4ed2-9475-7c34fb1af38b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:20.010711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 728/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:20.048487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf84cba-a309-4a6c-be65-272a7ae7b928 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:20.050149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5bf84cba-a309-4a6c-be65-272a7ae7b928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:20.062784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf84cba-a309-4a6c-be65-272a7ae7b928 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:20.066250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b987382e-5a7f-4147-a1e3-bba2f83043b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:20.066882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987382e-5a7f-4147-a1e3-bba2f83043b4 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:20.067168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987382e-5a7f-4147-a1e3-bba2f83043b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:20.067447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b987382e-5a7f-4147-a1e3-bba2f83043b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:20.076727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b987382e-5a7f-4147-a1e3-bba2f83043b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:20.078147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b987382e-5a7f-4147-a1e3-bba2f83043b4 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:20.087298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 733/1461] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:22.318437 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7222dda8-f979-40fc-83fc-40954074428a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:22.318974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7222dda8-f979-40fc-83fc-40954074428a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:22.318974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7222dda8-f979-40fc-83fc-40954074428a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:22.320346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 729/1462] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:22.326361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c420fe7c-a8e7-4a4f-ad25-cd6e7f0d8178 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:22.326785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c420fe7c-a8e7-4a4f-ad25-cd6e7f0d8178 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:22.326926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c420fe7c-a8e7-4a4f-ad25-cd6e7f0d8178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:22.333291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=79 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:04:22.345616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c420fe7c-a8e7-4a4f-ad25-cd6e7f0d8178 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:22.346813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c420fe7c-a8e7-4a4f-ad25-cd6e7f0d8178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:22.355086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 734/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:04:22.391624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7222dda8-f979-40fc-83fc-40954074428a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:22.397443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=73 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:04:22.404113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7222dda8-f979-40fc-83fc-40954074428a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:22.407880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2602f509-4672-45a8-9a78-7bec30510a97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:22.408559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602f509-4672-45a8-9a78-7bec30510a97 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:22.408873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602f509-4672-45a8-9a78-7bec30510a97 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:22.409291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2602f509-4672-45a8-9a78-7bec30510a97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:22.419126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2602f509-4672-45a8-9a78-7bec30510a97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:22.420578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2602f509-4672-45a8-9a78-7bec30510a97 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:22.429944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 730/1464] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:24.541597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f0ebd1-f472-4edf-830e-cfb2647ad698 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:24.542151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f0ebd1-f472-4edf-830e-cfb2647ad698 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:24.542151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f0ebd1-f472-4edf-830e-cfb2647ad698 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:24.543834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 735/1465] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:24.549021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6604aa41-26d8-4ff9-bb6d-aba02cd47e0e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:24.549448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6604aa41-26d8-4ff9-bb6d-aba02cd47e0e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:24.549602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6604aa41-26d8-4ff9-bb6d-aba02cd47e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:24.567372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6604aa41-26d8-4ff9-bb6d-aba02cd47e0e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:24.568596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6604aa41-26d8-4ff9-bb6d-aba02cd47e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:24.576864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 731/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:24.612432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f0ebd1-f472-4edf-830e-cfb2647ad698 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:24.624155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f0ebd1-f472-4edf-830e-cfb2647ad698 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:24.627625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5ab3d5-f226-4be9-9fc7-b89878c40dcd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:24.628257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ab3d5-f226-4be9-9fc7-b89878c40dcd service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:24.628582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ab3d5-f226-4be9-9fc7-b89878c40dcd service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:24.628865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ab3d5-f226-4be9-9fc7-b89878c40dcd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:24.637969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ab3d5-f226-4be9-9fc7-b89878c40dcd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:24.639397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ab3d5-f226-4be9-9fc7-b89878c40dcd service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:24.648705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 736/1467] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:26.691820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94bb62-3d07-4a42-bf4a-bbbd9d5ff2d2 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:26.692357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94bb62-3d07-4a42-bf4a-bbbd9d5ff2d2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:26.692357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94bb62-3d07-4a42-bf4a-bbbd9d5ff2d2 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:26.693764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 732/1468] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:04:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:26.699518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0069c1-6e0d-4cf8-bf2d-e1fc28495c40 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:26.699946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0069c1-6e0d-4cf8-bf2d-e1fc28495c40 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:26.700082 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0069c1-6e0d-4cf8-bf2d-e1fc28495c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:26.717975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-cc0069c1-6e0d-4cf8-bf2d-e1fc28495c40 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:26.719175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cc0069c1-6e0d-4cf8-bf2d-e1fc28495c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:26.727708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 737/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:04:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:26.763883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a94bb62-3d07-4a42-bf4a-bbbd9d5ff2d2 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:26.774573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a94bb62-3d07-4a42-bf4a-bbbd9d5ff2d2 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:26.778084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feedb6cc-345d-4f4b-b534-584ca05f2be4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:26.778751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feedb6cc-345d-4f4b-b534-584ca05f2be4 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:26.779037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feedb6cc-345d-4f4b-b534-584ca05f2be4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:26.779318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feedb6cc-345d-4f4b-b534-584ca05f2be4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:26.788113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feedb6cc-345d-4f4b-b534-584ca05f2be4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:26.789578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feedb6cc-345d-4f4b-b534-584ca05f2be4 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:26.799211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 733/1470] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:36.712431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=11 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:04:36.795063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=14 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:04:46.970350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f0208-9644-4e08-99d6-fe06d0d6bc5d None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:46.970896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f0208-9644-4e08-99d6-fe06d0d6bc5d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:46.970998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f0208-9644-4e08-99d6-fe06d0d6bc5d None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:46.972340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 738/1471] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:04:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:46.977410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f02d438-24ad-4c90-9e51-b2e5140901dc None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:46.977826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f02d438-24ad-4c90-9e51-b2e5140901dc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:46.977969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f02d438-24ad-4c90-9e51-b2e5140901dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:46.996047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0f02d438-24ad-4c90-9e51-b2e5140901dc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:46.997713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0f02d438-24ad-4c90-9e51-b2e5140901dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:47.005436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 734/1472] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:04:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:47.087857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a8ad5b-03b1-42b7-a160-2e94feb34807 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:47.088140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a8ad5b-03b1-42b7-a160-2e94feb34807 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:47.088442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a8ad5b-03b1-42b7-a160-2e94feb34807 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:47.089815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 739/1473] 10.222.0.20 () {54 vars in 962 bytes} [Tue Jul 5 10:04:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:47.094503 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85318fe-bf70-47d6-8bf3-4424a8ac2078 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:47.094792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85318fe-bf70-47d6-8bf3-4424a8ac2078 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:47.095070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85318fe-bf70-47d6-8bf3-4424a8ac2078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:47.113167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c85318fe-bf70-47d6-8bf3-4424a8ac2078 None None] MFA Rules not processed for user `90d9b9421d454465b8f99137cd182abf`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:47.114351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c85318fe-bf70-47d6-8bf3-4424a8ac2078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:47.120747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 735/1474] 10.222.0.20 () {58 vars in 1059 bytes} [Tue Jul 5 10:04:47 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:47.128481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a8ad5b-03b1-42b7-a160-2e94feb34807 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:47.139836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a8ad5b-03b1-42b7-a160-2e94feb34807 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:47.143293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597eec10-0fbd-45d9-8be8-86aaddd62a4e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:47.143928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597eec10-0fbd-45d9-8be8-86aaddd62a4e service glance] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:47.144210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597eec10-0fbd-45d9-8be8-86aaddd62a4e service glance] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:47.144513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597eec10-0fbd-45d9-8be8-86aaddd62a4e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:47.153334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597eec10-0fbd-45d9-8be8-86aaddd62a4e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:47.154748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597eec10-0fbd-45d9-8be8-86aaddd62a4e service glance] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:47.164078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 740/1475] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:04:49.255738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4886393-b714-4730-9d3e-45035ea61e7a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:49.256297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4886393-b714-4730-9d3e-45035ea61e7a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:49.256297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4886393-b714-4730-9d3e-45035ea61e7a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:49.257697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 736/1476] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:04:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:49.263591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cff3939-d1cc-4a6d-9f47-f3ee7d0ad80e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:49.263989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cff3939-d1cc-4a6d-9f47-f3ee7d0ad80e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:49.264148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cff3939-d1cc-4a6d-9f47-f3ee7d0ad80e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:49.281706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1cff3939-d1cc-4a6d-9f47-f3ee7d0ad80e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:49.282907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1cff3939-d1cc-4a6d-9f47-f3ee7d0ad80e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:49.290616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 741/1477] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:04:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:49.314582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4886393-b714-4730-9d3e-45035ea61e7a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:49.328197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4886393-b714-4730-9d3e-45035ea61e7a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:49.332690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d045003-8a4a-42a7-8226-536ee998e548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:49.333338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d045003-8a4a-42a7-8226-536ee998e548 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:49.333621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d045003-8a4a-42a7-8226-536ee998e548 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:49.333899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d045003-8a4a-42a7-8226-536ee998e548 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:49.334632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d045003-8a4a-42a7-8226-536ee998e548 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:49.336411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-0d045003-8a4a-42a7-8226-536ee998e548 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:04:49.336411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:04:49.336411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:04:49.337566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d045003-8a4a-42a7-8226-536ee998e548 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:49.356718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 737/1478] 10.222.0.20 () {56 vars in 1108 bytes} [Tue Jul 5 10:04:49 2022] GET /identity/v3/projects => generated 2553 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:04:51.392001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fe7fc3-4396-43b5-9193-e9c83a71fa09 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:51.392605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fe7fc3-4396-43b5-9193-e9c83a71fa09 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:51.392605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fe7fc3-4396-43b5-9193-e9c83a71fa09 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:51.393956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 742/1479] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:04:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:51.399134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5388c8b1-db4d-49b0-8250-af4ce8a729da None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:51.399529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5388c8b1-db4d-49b0-8250-af4ce8a729da None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:51.399723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5388c8b1-db4d-49b0-8250-af4ce8a729da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:51.417200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5388c8b1-db4d-49b0-8250-af4ce8a729da None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:51.418382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5388c8b1-db4d-49b0-8250-af4ce8a729da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:51.426303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 738/1480] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:04:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:51.469064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fe7fc3-4396-43b5-9193-e9c83a71fa09 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:51.483554 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fe7fc3-4396-43b5-9193-e9c83a71fa09 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:51.488892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3970ccca-dd01-482f-8815-58928524fc36 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:51.489531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3970ccca-dd01-482f-8815-58928524fc36 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:51.489816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3970ccca-dd01-482f-8815-58928524fc36 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:51.490092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3970ccca-dd01-482f-8815-58928524fc36 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:51.498585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3970ccca-dd01-482f-8815-58928524fc36 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:51.500166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3970ccca-dd01-482f-8815-58928524fc36 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:51.509445 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 743/1481] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:53.689854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ae5ca-9bfa-4f0c-86ad-e7c9e2800307 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:53.690851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ae5ca-9bfa-4f0c-86ad-e7c9e2800307 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:53.691140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6ae5ca-9bfa-4f0c-86ad-e7c9e2800307 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:53.692621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 739/1482] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:04:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:53.697723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41eaf63-9ca1-4677-8fd0-953bc405e0e6 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:53.698149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41eaf63-9ca1-4677-8fd0-953bc405e0e6 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:53.698288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41eaf63-9ca1-4677-8fd0-953bc405e0e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:53.716374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b41eaf63-9ca1-4677-8fd0-953bc405e0e6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:53.718184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b41eaf63-9ca1-4677-8fd0-953bc405e0e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:53.726248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 744/1483] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:04:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:53.761870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6ae5ca-9bfa-4f0c-86ad-e7c9e2800307 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:53.772896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6ae5ca-9bfa-4f0c-86ad-e7c9e2800307 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:53.776318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783a7666-cee8-4d9a-acbb-4c6d8bf05fef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:53.776982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a7666-cee8-4d9a-acbb-4c6d8bf05fef service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:53.777261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a7666-cee8-4d9a-acbb-4c6d8bf05fef service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:53.777550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a7666-cee8-4d9a-acbb-4c6d8bf05fef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:53.786842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783a7666-cee8-4d9a-acbb-4c6d8bf05fef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:53.788252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783a7666-cee8-4d9a-acbb-4c6d8bf05fef service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:53.797744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 740/1484] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:55.848993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94fd8c9-ff5f-4571-bb79-9e13d5ca42f7 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:55.849551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94fd8c9-ff5f-4571-bb79-9e13d5ca42f7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:55.849551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e94fd8c9-ff5f-4571-bb79-9e13d5ca42f7 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:55.850932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 745/1485] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:04:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:55.855963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ce49d-58fc-48c6-8ab8-bdc6ec404d82 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:55.856408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ce49d-58fc-48c6-8ab8-bdc6ec404d82 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:55.856667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1ce49d-58fc-48c6-8ab8-bdc6ec404d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:55.874338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-dc1ce49d-58fc-48c6-8ab8-bdc6ec404d82 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:55.875518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-dc1ce49d-58fc-48c6-8ab8-bdc6ec404d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:55.883623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 741/1486] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:04:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:55.920872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94fd8c9-ff5f-4571-bb79-9e13d5ca42f7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:55.931689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e94fd8c9-ff5f-4571-bb79-9e13d5ca42f7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:55.935776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b2487-5908-42ce-b3a2-a9597ea7d521 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:55.936427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b2487-5908-42ce-b3a2-a9597ea7d521 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:55.936713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b2487-5908-42ce-b3a2-a9597ea7d521 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:55.936984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b2487-5908-42ce-b3a2-a9597ea7d521 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:55.945809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b2487-5908-42ce-b3a2-a9597ea7d521 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:55.947243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b2487-5908-42ce-b3a2-a9597ea7d521 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:55.957355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 746/1487] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:04:58.159793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b6f804-0f3e-44ba-a78d-2b771c06bcc2 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:58.160346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b6f804-0f3e-44ba-a78d-2b771c06bcc2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:58.160346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b6f804-0f3e-44ba-a78d-2b771c06bcc2 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:58.161740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 742/1488] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:04:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:04:58.167014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d99296d-977d-494f-8596-2eb1044f9adc None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:58.167423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d99296d-977d-494f-8596-2eb1044f9adc None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:58.167574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d99296d-977d-494f-8596-2eb1044f9adc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:58.185798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0d99296d-977d-494f-8596-2eb1044f9adc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:04:58.186999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0d99296d-977d-494f-8596-2eb1044f9adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:04:58.194691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 747/1489] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:04:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:04:58.234920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b6f804-0f3e-44ba-a78d-2b771c06bcc2 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:04:58.245483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b6f804-0f3e-44ba-a78d-2b771c06bcc2 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:04:58.248969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcdc2a9-2cc5-4207-87bf-9a760e28d806 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:04:58.249606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcdc2a9-2cc5-4207-87bf-9a760e28d806 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:04:58.249889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcdc2a9-2cc5-4207-87bf-9a760e28d806 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:04:58.250167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcdc2a9-2cc5-4207-87bf-9a760e28d806 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:04:58.258882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcdc2a9-2cc5-4207-87bf-9a760e28d806 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:04:58.260327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcdc2a9-2cc5-4207-87bf-9a760e28d806 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:04:58.270139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 743/1490] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:04:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:05:00.451073 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2614871-9b61-4750-98a2-a5c8c7078514 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:05:00.451552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2614871-9b61-4750-98a2-a5c8c7078514 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:05:00.451669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2614871-9b61-4750-98a2-a5c8c7078514 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:05:00.453083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 748/1491] 10.222.0.20 () {54 vars in 926 bytes} [Tue Jul 5 10:05:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:05:00.458140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee36f03f-eadc-423e-9017-a7011bf3804b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:05:00.458576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee36f03f-eadc-423e-9017-a7011bf3804b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:05:00.458686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee36f03f-eadc-423e-9017-a7011bf3804b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:05:00.475857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ee36f03f-eadc-423e-9017-a7011bf3804b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:05:00.477065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ee36f03f-eadc-423e-9017-a7011bf3804b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:05:00.484665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 744/1492] 10.222.0.20 () {58 vars in 1023 bytes} [Tue Jul 5 10:05:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:05:00.523129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2614871-9b61-4750-98a2-a5c8c7078514 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:05:00.533563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2614871-9b61-4750-98a2-a5c8c7078514 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:05:00.537011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca185fe-fd31-4b17-9235-f646a0b6fe8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:05:00.537637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca185fe-fd31-4b17-9235-f646a0b6fe8c service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:05:00.537922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca185fe-fd31-4b17-9235-f646a0b6fe8c service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:05:00.538197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca185fe-fd31-4b17-9235-f646a0b6fe8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:05:00.547731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca185fe-fd31-4b17-9235-f646a0b6fe8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:05:00.549253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca185fe-fd31-4b17-9235-f646a0b6fe8c service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:05:00.558503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 749/1493] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:05:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:05:10.471013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=33 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:05:10.554305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=33 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:06:25.401127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd7f737-4f7f-47fd-bf7a-4fdc681cddc8 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.401715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd7f737-4f7f-47fd-bf7a-4fdc681cddc8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.401715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd7f737-4f7f-47fd-bf7a-4fdc681cddc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.419779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ccd7f737-4f7f-47fd-bf7a-4fdc681cddc8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:06:25.421031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ccd7f737-4f7f-47fd-bf7a-4fdc681cddc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:06:25.429042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 745/1494] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:06:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:06:25.436744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e17137-06f6-4fe6-bb30-5cc16720cf47 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.437204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e17137-06f6-4fe6-bb30-5cc16720cf47 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.437314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e17137-06f6-4fe6-bb30-5cc16720cf47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.455324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c9e17137-06f6-4fe6-bb30-5cc16720cf47 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:06:25.456543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c9e17137-06f6-4fe6-bb30-5cc16720cf47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:06:25.464231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 750/1495] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:06:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:06:25.468647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd7f737-4f7f-47fd-bf7a-4fdc681cddc8 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:25.478734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd7f737-4f7f-47fd-bf7a-4fdc681cddc8 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:25.483203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:25.483844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.484127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.484420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.485152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:06:25.485463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:25.486474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:25.486474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:06:25.486474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:06:25.487362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:25.498629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 746/1496] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:06:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:06:25.506317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e17137-06f6-4fe6-bb30-5cc16720cf47 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:25.515297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e17137-06f6-4fe6-bb30-5cc16720cf47 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:25.520271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:25.520944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.521227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.521500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.522185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:25.523909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING py.warnings [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:25.523909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: warnings.warn(msg) Jul 05 10:06:25.523909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: Jul 05 10:06:25.525146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:25.560941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 751/1497] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:06:25 2022] POST /identity/v3/projects => generated 379 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:06:25.565155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:25.574856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b84d2ca-0d50-41d9-baa1-8294a9634b69 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:25.630128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:25.630792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.631077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.631354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.632029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:25.633069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:25.633069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:06:25.633069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:06:25.634284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:25.653548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 747/1498] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:06:25 2022] POST /identity/v3/users => generated 290 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:06:25.658129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:25.667977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda74ec7-29e6-4062-9d56-d2f0fae29188 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:25.672451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:25.673109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.673456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.673738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.674459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:25.675465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING py.warnings [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:25.675465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: warnings.warn(msg) Jul 05 10:06:25.675465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: Jul 05 10:06:25.676346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:25.689475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 752/1499] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:06:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:06:25.693282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:25.703326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb04999-1251-4a1d-81e1-27dcc830750a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:25.708366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696903e0-1adb-4fb1-ac10-0bb6e35c4e8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:25.709048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696903e0-1adb-4fb1-ac10-0bb6e35c4e8f admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.709332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696903e0-1adb-4fb1-ac10-0bb6e35c4e8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.709609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696903e0-1adb-4fb1-ac10-0bb6e35c4e8f admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.710326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696903e0-1adb-4fb1-ac10-0bb6e35c4e8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:25.711330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-696903e0-1adb-4fb1-ac10-0bb6e35c4e8f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:25.711330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:06:25.711330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:06:25.712205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696903e0-1adb-4fb1-ac10-0bb6e35c4e8f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:25.724771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 748/1500] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:06:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:06:25.728745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:25.738362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b3571e-b880-4bc3-a8df-2954e6d2bc32 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:25.742790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88804258-9475-4c06-83fe-1c2c1112e7c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:25.743657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88804258-9475-4c06-83fe-1c2c1112e7c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.743941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88804258-9475-4c06-83fe-1c2c1112e7c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.744216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88804258-9475-4c06-83fe-1c2c1112e7c4 admin admin] PATH_INFO: `/v3/projects/335f26d438834797a483ae0ccae87b3c/users/02d4e976f6a24a5cba5fbb5475c75284/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.758763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88804258-9475-4c06-83fe-1c2c1112e7c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=335f26d438834797a483ae0ccae87b3c, user_id=02d4e976f6a24a5cba5fbb5475c75284, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:25.759778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING py.warnings [None req-88804258-9475-4c06-83fe-1c2c1112e7c4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:25.759778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: warnings.warn(msg) Jul 05 10:06:25.759778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: Jul 05 10:06:25.761741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88804258-9475-4c06-83fe-1c2c1112e7c4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:25.774690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 753/1501] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:06:25 2022] PUT /identity/v3/projects/335f26d438834797a483ae0ccae87b3c/users/02d4e976f6a24a5cba5fbb5475c75284/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:06:25.786405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d365f3-1e6d-4004-a62e-fc551e461aae None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.786755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d365f3-1e6d-4004-a62e-fc551e461aae None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.787039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d365f3-1e6d-4004-a62e-fc551e461aae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.813786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-08d365f3-1e6d-4004-a62e-fc551e461aae None None] MFA Rules not processed for user `02d4e976f6a24a5cba5fbb5475c75284`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:06:25.815387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-08d365f3-1e6d-4004-a62e-fc551e461aae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:06:25.877628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 749/1502] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:06:25 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:06:25.890527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae4b919-d021-4257-8a1e-c56fa618ff11 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.890818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae4b919-d021-4257-8a1e-c56fa618ff11 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.891728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae4b919-d021-4257-8a1e-c56fa618ff11 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.893202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 754/1503] 10.222.0.20 () {54 vars in 963 bytes} [Tue Jul 5 10:06:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:06:25.897777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42c119d-104a-450f-8fc8-6effbf54cc46 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.898070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42c119d-104a-450f-8fc8-6effbf54cc46 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.898351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42c119d-104a-450f-8fc8-6effbf54cc46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:25.915800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e42c119d-104a-450f-8fc8-6effbf54cc46 None None] MFA Rules not processed for user `1fc5b3da67de48cb83848168b16dd153`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:06:25.916966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e42c119d-104a-450f-8fc8-6effbf54cc46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:06:25.962290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:06:25.970694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 750/1504] 10.222.0.20 () {58 vars in 1060 bytes} [Tue Jul 5 10:06:25 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:06:25.978341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae4b919-d021-4257-8a1e-c56fa618ff11 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:25.988790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae4b919-d021-4257-8a1e-c56fa618ff11 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:25.992279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:25.992969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:25.993258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:25.993544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:26.002239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:26.003669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:26.012463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 755/1505] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:06:25 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:06:26.053728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c31f5-41dc-4d26-93c2-cf36bf9463ad None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:26.054018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c31f5-41dc-4d26-93c2-cf36bf9463ad None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:26.054298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c31f5-41dc-4d26-93c2-cf36bf9463ad None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:26.055577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 751/1506] 10.222.0.20 () {52 vars in 829 bytes} [Tue Jul 5 10:06:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:06:26.060768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:26.121627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c897a21-4286-4f50-869a-069c6a91f0ec service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:26.127032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:26.127904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:26.128193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:26.128503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] PATH_INFO: `/v3/users/02d4e976f6a24a5cba5fbb5475c75284` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:26.129688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02d4e976f6a24a5cba5fbb5475c75284)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:26.130705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING py.warnings [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:26.130705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: warnings.warn(msg) Jul 05 10:06:26.130705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: Jul 05 10:06:26.131924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:26.196982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02d4e976f6a24a5cba5fbb5475c75284'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.209409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02d4e976f6a24a5cba5fbb5475c75284'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.210233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02d4e976f6a24a5cba5fbb5475c75284'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.210884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02d4e976f6a24a5cba5fbb5475c75284'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.228957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02d4e976f6a24a5cba5fbb5475c75284'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.232935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:06:26.245282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 756/1507] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:06:26 2022] DELETE /identity/v3/users/02d4e976f6a24a5cba5fbb5475c75284 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:06:26.249810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:26.250102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:26.250969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:26.267801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:06:26.269032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:06:26.327241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 752/1508] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:06:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:06:26.336701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:26.338149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:06:26.396211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce81e61-3578-4ac3-ab36-34e8f4675858 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:26.399784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6de0dc1-2dcf-4dea-8d0d-fb2c96940af8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:26.400458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6de0dc1-2dcf-4dea-8d0d-fb2c96940af8 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:26.400748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6de0dc1-2dcf-4dea-8d0d-fb2c96940af8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:26.401048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6de0dc1-2dcf-4dea-8d0d-fb2c96940af8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:26.410272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6de0dc1-2dcf-4dea-8d0d-fb2c96940af8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:26.411707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6de0dc1-2dcf-4dea-8d0d-fb2c96940af8 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:26.420972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 757/1509] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:06:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:06:26.772929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:06:26.774515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:06:26.787066 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c85c249-6888-4bc2-b046-0eb1d0817da9 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:06:26.792124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:06:26.793006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:06:26.793293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:06:26.793569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] PATH_INFO: `/v3/projects/335f26d438834797a483ae0ccae87b3c` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:06:26.796595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=335f26d438834797a483ae0ccae87b3c)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:06:26.796595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:06:26.796595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:06:26.796595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:06:26.797187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:06:26.859910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '335f26d438834797a483ae0ccae87b3c', 'request_id': 'req-6d40f454-7d46-4282-9a07-0346074a1a9e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.865969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '335f26d438834797a483ae0ccae87b3c', 'request_id': 'req-6d40f454-7d46-4282-9a07-0346074a1a9e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.875488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '335f26d438834797a483ae0ccae87b3c', 'request_id': 'req-6d40f454-7d46-4282-9a07-0346074a1a9e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.894177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] The token cache is being invalidate because project 335f26d438834797a483ae0ccae87b3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:06:26.894459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.895101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6d40f454-7d46-4282-9a07-0346074a1a9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:06:26.896779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 753/1510] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:06:26 2022] DELETE /identity/v3/projects/335f26d438834797a483ae0ccae87b3c => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:06:36.416847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=1,keystone:SELECT=28 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:06:36.890542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=55,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:08:03.912457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072ebf9-62b0-4d3b-9588-775c63f58e1f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:03.913055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072ebf9-62b0-4d3b-9588-775c63f58e1f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:03.913055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072ebf9-62b0-4d3b-9588-775c63f58e1f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:03.914407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 758/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:08:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:03.919236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8b75b9-145b-421b-aeca-81c6423c1859 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:03.919679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8b75b9-145b-421b-aeca-81c6423c1859 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:03.919786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8b75b9-145b-421b-aeca-81c6423c1859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:03.938744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7e8b75b9-145b-421b-aeca-81c6423c1859 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:03.939953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7e8b75b9-145b-421b-aeca-81c6423c1859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:03.997120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 754/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:08:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:04.151080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072ebf9-62b0-4d3b-9588-775c63f58e1f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:04.152789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b072ebf9-62b0-4d3b-9588-775c63f58e1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:04.225826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072ebf9-62b0-4d3b-9588-775c63f58e1f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:04.231168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ade2a93-aad6-46b0-bd40-711d07940881 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:04.231817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ade2a93-aad6-46b0-bd40-711d07940881 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:04.232103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ade2a93-aad6-46b0-bd40-711d07940881 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:04.232384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ade2a93-aad6-46b0-bd40-711d07940881 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:04.240483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ade2a93-aad6-46b0-bd40-711d07940881 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:04.242086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ade2a93-aad6-46b0-bd40-711d07940881 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:04.251311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 759/1513] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:05.074015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a702cef5-cfc8-4bd7-ac08-0f97040f8719 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:05.074560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a702cef5-cfc8-4bd7-ac08-0f97040f8719 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:05.074656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a702cef5-cfc8-4bd7-ac08-0f97040f8719 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:05.075993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 755/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:08:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:05.080826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28881301-2211-4bbf-abd8-f77cc525ec8c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:05.081250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28881301-2211-4bbf-abd8-f77cc525ec8c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:05.081396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28881301-2211-4bbf-abd8-f77cc525ec8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:05.099587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-28881301-2211-4bbf-abd8-f77cc525ec8c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:05.100884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-28881301-2211-4bbf-abd8-f77cc525ec8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:05.109211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 760/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:08:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:05.259707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a702cef5-cfc8-4bd7-ac08-0f97040f8719 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:05.274108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a702cef5-cfc8-4bd7-ac08-0f97040f8719 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:05.279500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-004b9edb-9b1b-4c32-b1b7-cd6cb4815bdc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:05.280136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004b9edb-9b1b-4c32-b1b7-cd6cb4815bdc service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:05.280450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004b9edb-9b1b-4c32-b1b7-cd6cb4815bdc service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:05.280732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-004b9edb-9b1b-4c32-b1b7-cd6cb4815bdc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:05.288734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004b9edb-9b1b-4c32-b1b7-cd6cb4815bdc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:05.290837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-004b9edb-9b1b-4c32-b1b7-cd6cb4815bdc service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:05.300145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 756/1516] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:06.074988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f3ec2-7006-4b25-b6de-70cb9e1107ba None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:06.075272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f3ec2-7006-4b25-b6de-70cb9e1107ba None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:06.075547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202f3ec2-7006-4b25-b6de-70cb9e1107ba None None] PATH_INFO: `/v3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:06.076924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 761/1517] 10.222.0.30 () {54 vars in 925 bytes} [Tue Jul 5 10:08:06 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:08:06.083072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9070ec6f-bbff-4b9d-adc6-1adca20e0e25 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:06.083531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9070ec6f-bbff-4b9d-adc6-1adca20e0e25 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:06.083651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9070ec6f-bbff-4b9d-adc6-1adca20e0e25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:06.102172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9070ec6f-bbff-4b9d-adc6-1adca20e0e25 None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:06.103383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9070ec6f-bbff-4b9d-adc6-1adca20e0e25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:06.164662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 757/1518] 10.222.0.30 () {58 vars in 1015 bytes} [Tue Jul 5 10:08:06 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:15.094186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=37 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:08:15.250770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8041809c-24c9-4ae0-a489-29bc9f05019b None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:15.251061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8041809c-24c9-4ae0-a489-29bc9f05019b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:15.251342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8041809c-24c9-4ae0-a489-29bc9f05019b None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:15.252829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 762/1519] 10.222.0.20 () {54 vars in 960 bytes} [Tue Jul 5 10:08:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:15.257707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b46906-33d7-424d-bb63-09d486a40386 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:15.258194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b46906-33d7-424d-bb63-09d486a40386 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:15.258298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b46906-33d7-424d-bb63-09d486a40386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:15.276949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-25b46906-33d7-424d-bb63-09d486a40386 None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:15.278161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-25b46906-33d7-424d-bb63-09d486a40386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:15.286471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 758/1520] 10.222.0.20 () {58 vars in 1057 bytes} [Tue Jul 5 10:08:15 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:15.293848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8041809c-24c9-4ae0-a489-29bc9f05019b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:15.307411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8041809c-24c9-4ae0-a489-29bc9f05019b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:15.312593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ce5786-2a55-468f-bea7-4f073cb1fa81 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:15.313251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce5786-2a55-468f-bea7-4f073cb1fa81 service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:15.313538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce5786-2a55-468f-bea7-4f073cb1fa81 service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:15.313819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ce5786-2a55-468f-bea7-4f073cb1fa81 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:15.321960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ce5786-2a55-468f-bea7-4f073cb1fa81 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:15.324176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ce5786-2a55-468f-bea7-4f073cb1fa81 service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:15.333704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 763/1521] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:16.540312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cdf2f-6423-4f19-aebc-c9e3a4d6a805 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:16.540789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cdf2f-6423-4f19-aebc-c9e3a4d6a805 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:16.540901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cdf2f-6423-4f19-aebc-c9e3a4d6a805 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:16.542294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 759/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:08:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:16.546910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa507b-8761-4502-983a-129baf51f5bd None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:16.547292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa507b-8761-4502-983a-129baf51f5bd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:16.547475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa507b-8761-4502-983a-129baf51f5bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:16.565853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-63fa507b-8761-4502-983a-129baf51f5bd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:16.567049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-63fa507b-8761-4502-983a-129baf51f5bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:16.575365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 764/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:08:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:16.727611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023cdf2f-6423-4f19-aebc-c9e3a4d6a805 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:16.742667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023cdf2f-6423-4f19-aebc-c9e3a4d6a805 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:16.748126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b789bd16-8f1c-4344-ab4f-6c482cf4abd8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:16.748890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b789bd16-8f1c-4344-ab4f-6c482cf4abd8 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:16.749092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b789bd16-8f1c-4344-ab4f-6c482cf4abd8 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:16.749377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b789bd16-8f1c-4344-ab4f-6c482cf4abd8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:16.757850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b789bd16-8f1c-4344-ab4f-6c482cf4abd8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:16.759450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b789bd16-8f1c-4344-ab4f-6c482cf4abd8 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:16.768834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 760/1524] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:17.578048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ed7ea5-d246-478d-a0eb-f579e96fb963 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:17.578615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ed7ea5-d246-478d-a0eb-f579e96fb963 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:17.578711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ed7ea5-d246-478d-a0eb-f579e96fb963 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:17.580046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 765/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 5 10:08:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:17.584828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1042be-324e-4f97-a909-53eab7f52120 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:17.585272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1042be-324e-4f97-a909-53eab7f52120 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:17.585400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1042be-324e-4f97-a909-53eab7f52120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:17.603621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-1d1042be-324e-4f97-a909-53eab7f52120 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:17.605414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1d1042be-324e-4f97-a909-53eab7f52120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:17.613139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 761/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 5 10:08:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:17.763654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ed7ea5-d246-478d-a0eb-f579e96fb963 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:17.778645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ed7ea5-d246-478d-a0eb-f579e96fb963 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:17.784017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a7aae4-4634-43fa-aaec-91a295e47148 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:17.784692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a7aae4-4634-43fa-aaec-91a295e47148 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:17.784982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a7aae4-4634-43fa-aaec-91a295e47148 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:17.785265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a7aae4-4634-43fa-aaec-91a295e47148 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:17.793379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a7aae4-4634-43fa-aaec-91a295e47148 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:17.794961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a7aae4-4634-43fa-aaec-91a295e47148 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:17.804370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 766/1527] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:21.610673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624c82f-d3b0-4ba4-a9d6-c7e8ea5d18e8 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:21.611233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624c82f-d3b0-4ba4-a9d6-c7e8ea5d18e8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:21.611233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624c82f-d3b0-4ba4-a9d6-c7e8ea5d18e8 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:21.612629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 762/1528] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:08:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:21.617873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094e79f0-dab4-4168-83f3-38086b3fe801 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:21.618258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094e79f0-dab4-4168-83f3-38086b3fe801 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:21.618433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094e79f0-dab4-4168-83f3-38086b3fe801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:21.636632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-094e79f0-dab4-4168-83f3-38086b3fe801 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:21.637839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-094e79f0-dab4-4168-83f3-38086b3fe801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:21.645615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 767/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:08:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:21.718548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624c82f-d3b0-4ba4-a9d6-c7e8ea5d18e8 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:21.720239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9624c82f-d3b0-4ba4-a9d6-c7e8ea5d18e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:21.777675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624c82f-d3b0-4ba4-a9d6-c7e8ea5d18e8 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:21.781193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa52a61-0e07-46c1-8f8b-416131b1ad61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:21.781828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa52a61-0e07-46c1-8f8b-416131b1ad61 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:21.782107 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa52a61-0e07-46c1-8f8b-416131b1ad61 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:21.782377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa52a61-0e07-46c1-8f8b-416131b1ad61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:21.791079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa52a61-0e07-46c1-8f8b-416131b1ad61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:21.792548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa52a61-0e07-46c1-8f8b-416131b1ad61 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:21.797372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:08:21.802418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 763/1530] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:23.798916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f370d4be-64ca-4908-94c9-b0c0107c4c4e None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:23.799475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f370d4be-64ca-4908-94c9-b0c0107c4c4e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:23.799475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f370d4be-64ca-4908-94c9-b0c0107c4c4e None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:23.801058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 768/1531] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:08:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:23.805919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079c849-57aa-41f6-9876-2cae2677c60c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:23.806359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079c849-57aa-41f6-9876-2cae2677c60c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:23.806456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7079c849-57aa-41f6-9876-2cae2677c60c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:23.825638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7079c849-57aa-41f6-9876-2cae2677c60c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:23.826843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7079c849-57aa-41f6-9876-2cae2677c60c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:23.834597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 764/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:08:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:23.870092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f370d4be-64ca-4908-94c9-b0c0107c4c4e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:23.872260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f370d4be-64ca-4908-94c9-b0c0107c4c4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:23.884899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f370d4be-64ca-4908-94c9-b0c0107c4c4e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:23.888640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e7c64c-b937-4697-bdb2-548873537488 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:23.889288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e7c64c-b937-4697-bdb2-548873537488 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:23.889573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e7c64c-b937-4697-bdb2-548873537488 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:23.889849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e7c64c-b937-4697-bdb2-548873537488 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:23.898819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e7c64c-b937-4697-bdb2-548873537488 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:23.900277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e7c64c-b937-4697-bdb2-548873537488 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:23.909702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 769/1533] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:27.454835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0cd249-1340-4b0d-b9e8-748e86c31f0a None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:27.455404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0cd249-1340-4b0d-b9e8-748e86c31f0a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:27.455404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0cd249-1340-4b0d-b9e8-748e86c31f0a None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:27.456849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 765/1534] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:08:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:27.462168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c85df-6b86-4399-901f-4b638bba88a4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:27.462591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c85df-6b86-4399-901f-4b638bba88a4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:27.462716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c85df-6b86-4399-901f-4b638bba88a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:27.480249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5d3c85df-6b86-4399-901f-4b638bba88a4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:27.482011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5d3c85df-6b86-4399-901f-4b638bba88a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:27.489745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 770/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:08:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:27.788801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0cd249-1340-4b0d-b9e8-748e86c31f0a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:27.790541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0c0cd249-1340-4b0d-b9e8-748e86c31f0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:27.846821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0cd249-1340-4b0d-b9e8-748e86c31f0a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:27.850314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275dab69-d8b5-45e0-a774-6f29c1e6d1c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:27.850947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275dab69-d8b5-45e0-a774-6f29c1e6d1c3 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:27.851230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275dab69-d8b5-45e0-a774-6f29c1e6d1c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:27.851504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275dab69-d8b5-45e0-a774-6f29c1e6d1c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:27.860860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275dab69-d8b5-45e0-a774-6f29c1e6d1c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:27.862273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275dab69-d8b5-45e0-a774-6f29c1e6d1c3 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:27.871440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 766/1536] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:29.891297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e853079c-b816-4f33-a899-e91b51468cf6 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:29.891869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e853079c-b816-4f33-a899-e91b51468cf6 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:29.891869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e853079c-b816-4f33-a899-e91b51468cf6 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:29.893273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 771/1537] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 5 10:08:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:08:29.898323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f3cdb5-18da-4b49-b801-ad0b573a3051 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:29.898712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f3cdb5-18da-4b49-b801-ad0b573a3051 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:29.898895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f3cdb5-18da-4b49-b801-ad0b573a3051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:29.916729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b3f3cdb5-18da-4b49-b801-ad0b573a3051 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:08:29.917924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b3f3cdb5-18da-4b49-b801-ad0b573a3051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:29.926385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 767/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 5 10:08:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:08:30.225932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e853079c-b816-4f33-a899-e91b51468cf6 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:08:30.227605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e853079c-b816-4f33-a899-e91b51468cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:08:30.241037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e853079c-b816-4f33-a899-e91b51468cf6 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:08:30.244531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e0c93a-b286-4bf8-a217-02dc53216ad2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:08:30.245171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e0c93a-b286-4bf8-a217-02dc53216ad2 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:08:30.245453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e0c93a-b286-4bf8-a217-02dc53216ad2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:08:30.245733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e0c93a-b286-4bf8-a217-02dc53216ad2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:08:30.254555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e0c93a-b286-4bf8-a217-02dc53216ad2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:08:30.255970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e0c93a-b286-4bf8-a217-02dc53216ad2 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:08:30.265361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 772/1539] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:08:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:08:39.911576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=32 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:08:40.261064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=33 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:01.451199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c968ee-3e41-49f4-b235-734f293e6e6a None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.454404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c968ee-3e41-49f4-b235-734f293e6e6a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.454404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c968ee-3e41-49f4-b235-734f293e6e6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.474575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-79c968ee-3e41-49f4-b235-734f293e6e6a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.475896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-79c968ee-3e41-49f4-b235-734f293e6e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.487510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 768/1540] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.497006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e60c9-3263-4047-9d2b-fa3084d2c1f4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.497562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e60c9-3263-4047-9d2b-fa3084d2c1f4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.497562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e60c9-3263-4047-9d2b-fa3084d2c1f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.523371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-3f6e60c9-3263-4047-9d2b-fa3084d2c1f4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.524771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3f6e60c9-3263-4047-9d2b-fa3084d2c1f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.538179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 773/1541] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.542529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6784bbf7-e09b-4997-8ac0-6586a1d16d7e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.542704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6e60c9-3263-4047-9d2b-fa3084d2c1f4 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:01.543249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6784bbf7-e09b-4997-8ac0-6586a1d16d7e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.543743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6784bbf7-e09b-4997-8ac0-6586a1d16d7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.553410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6e60c9-3263-4047-9d2b-fa3084d2c1f4 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:01.565406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6784bbf7-e09b-4997-8ac0-6586a1d16d7e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.566540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:01.567271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.567556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.567836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.568164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6784bbf7-e09b-4997-8ac0-6586a1d16d7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.568600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:01.569172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:01.570188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING py.warnings [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:10:01.570188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: warnings.warn(msg) Jul 05 10:10:01.570188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: Jul 05 10:10:01.575254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:01.589051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 769/1542] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.591321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 774/1543] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:10:01.599422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e9556a-4d6a-438e-bb6d-67d8e8c7afc7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.599700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:01.600356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e9556a-4d6a-438e-bb6d-67d8e8c7afc7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.600869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e9556a-4d6a-438e-bb6d-67d8e8c7afc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.616419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d3bec2-0e2c-4f19-8567-b9176c38abf7 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:01.624000 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:01.624702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.624984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.625259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.625948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:01.630457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:01.631411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-70e9556a-4d6a-438e-bb6d-67d8e8c7afc7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.632938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-70e9556a-4d6a-438e-bb6d-67d8e8c7afc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.647375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 770/1544] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.657616 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e9556a-4d6a-438e-bb6d-67d8e8c7afc7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:01.674059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e9556a-4d6a-438e-bb6d-67d8e8c7afc7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:01.684735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 775/1545] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/projects => generated 385 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:01.684735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:01.767320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c7536f-d2cf-4b92-a632-8d56c9127535 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:01.783526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebcee9c-fcc2-4c5c-9a86-1df85a78c802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:01.784274 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebcee9c-fcc2-4c5c-9a86-1df85a78c802 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.784589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebcee9c-fcc2-4c5c-9a86-1df85a78c802 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.784866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebcee9c-fcc2-4c5c-9a86-1df85a78c802 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.786356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebcee9c-fcc2-4c5c-9a86-1df85a78c802 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:01.788861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING py.warnings [None req-9ebcee9c-fcc2-4c5c-9a86-1df85a78c802 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:10:01.788861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: warnings.warn(msg) Jul 05 10:10:01.788861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: Jul 05 10:10:01.790157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebcee9c-fcc2-4c5c-9a86-1df85a78c802 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:01.801137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffa62f3-d78a-4e20-8db8-a5270683c682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:01.801822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffa62f3-d78a-4e20-8db8-a5270683c682 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.802161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffa62f3-d78a-4e20-8db8-a5270683c682 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.802442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffa62f3-d78a-4e20-8db8-a5270683c682 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.803188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffa62f3-d78a-4e20-8db8-a5270683c682 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:01.803500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffa62f3-d78a-4e20-8db8-a5270683c682 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:01.805422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffa62f3-d78a-4e20-8db8-a5270683c682 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:01.821411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 771/1546] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:10:01.822835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01426ffe-3ceb-4c4c-9e52-a8a816a55ae8 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.823129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01426ffe-3ceb-4c4c-9e52-a8a816a55ae8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.823416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01426ffe-3ceb-4c4c-9e52-a8a816a55ae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.825853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 776/1547] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/users => generated 293 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:01.827223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff849100-ec90-42cb-a98f-a4809f0d44e9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.827526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff849100-ec90-42cb-a98f-a4809f0d44e9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.827954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff849100-ec90-42cb-a98f-a4809f0d44e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.847604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ff849100-ec90-42cb-a98f-a4809f0d44e9 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.849393 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-01426ffe-3ceb-4c4c-9e52-a8a816a55ae8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.849532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ff849100-ec90-42cb-a98f-a4809f0d44e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.850582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-01426ffe-3ceb-4c4c-9e52-a8a816a55ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.857407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 777/1548] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.858719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0b599-d92c-4a90-ae63-430c95dfd9ff None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.859023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0b599-d92c-4a90-ae63-430c95dfd9ff None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.859315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d0b599-d92c-4a90-ae63-430c95dfd9ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.859431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 772/1549] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.860751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b903ff9-c333-4947-978d-2962d93601d0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.861039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b903ff9-c333-4947-978d-2962d93601d0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.861377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b903ff9-c333-4947-978d-2962d93601d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.880135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-69d0b599-d92c-4a90-ae63-430c95dfd9ff None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.881432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-69d0b599-d92c-4a90-ae63-430c95dfd9ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.884980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-3b903ff9-c333-4947-978d-2962d93601d0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.886127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3b903ff9-c333-4947-978d-2962d93601d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.889265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 778/1550] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.890479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53cdb8d-d576-4a67-a9ed-de9c40d595f9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.890771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53cdb8d-d576-4a67-a9ed-de9c40d595f9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.891051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53cdb8d-d576-4a67-a9ed-de9c40d595f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.900596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 773/1551] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.901623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b903ff9-c333-4947-978d-2962d93601d0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:01.909936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e53cdb8d-d576-4a67-a9ed-de9c40d595f9 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.911728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e53cdb8d-d576-4a67-a9ed-de9c40d595f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.911947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b903ff9-c333-4947-978d-2962d93601d0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:01.916432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80244829-5d46-452c-a8ae-b26d0653e60e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:01.917109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80244829-5d46-452c-a8ae-b26d0653e60e admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.917391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80244829-5d46-452c-a8ae-b26d0653e60e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.917670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80244829-5d46-452c-a8ae-b26d0653e60e admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.918390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80244829-5d46-452c-a8ae-b26d0653e60e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:01.923321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 779/1552] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.923486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80244829-5d46-452c-a8ae-b26d0653e60e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:01.923607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53cdb8d-d576-4a67-a9ed-de9c40d595f9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:01.934410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53cdb8d-d576-4a67-a9ed-de9c40d595f9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:01.938881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 774/1553] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:01 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:01.940059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f145b1-eb8c-4e24-a2a9-473e14470cd7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.940384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f145b1-eb8c-4e24-a2a9-473e14470cd7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.940706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f145b1-eb8c-4e24-a2a9-473e14470cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.942208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48257793-53fa-4809-a25a-58e8d190692a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:01.942927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48257793-53fa-4809-a25a-58e8d190692a admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.943145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48257793-53fa-4809-a25a-58e8d190692a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.943557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48257793-53fa-4809-a25a-58e8d190692a admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.945032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48257793-53fa-4809-a25a-58e8d190692a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:01.946884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48257793-53fa-4809-a25a-58e8d190692a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:01.959501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-46f145b1-eb8c-4e24-a2a9-473e14470cd7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.960745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-46f145b1-eb8c-4e24-a2a9-473e14470cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:01.970921 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 775/1554] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:01.974837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 780/1555] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/projects => generated 377 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:01.976171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a384e305-40e8-4b47-a1f5-ca1b0b3b091c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.976587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a384e305-40e8-4b47-a1f5-ca1b0b3b091c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.977079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a384e305-40e8-4b47-a1f5-ca1b0b3b091c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.978284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59fe3e5-98b9-45a1-8bcd-1083f0162c8e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:01.978581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59fe3e5-98b9-45a1-8bcd-1083f0162c8e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:01.979224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59fe3e5-98b9-45a1-8bcd-1083f0162c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:01.997556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a59fe3e5-98b9-45a1-8bcd-1083f0162c8e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:01.998794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a59fe3e5-98b9-45a1-8bcd-1083f0162c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:02.063885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 776/1556] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:02.065521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88354c2-2856-4cb3-bda7-cc61fb6c1b3b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.065984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88354c2-2856-4cb3-bda7-cc61fb6c1b3b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.066423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88354c2-2856-4cb3-bda7-cc61fb6c1b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.085975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b88354c2-2856-4cb3-bda7-cc61fb6c1b3b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:02.087201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b88354c2-2856-4cb3-bda7-cc61fb6c1b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:02.095148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 777/1557] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:02.096467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab81a8b8-2c97-4b96-81bf-d3e8c47f774c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.096755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab81a8b8-2c97-4b96-81bf-d3e8c47f774c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.097028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab81a8b8-2c97-4b96-81bf-d3e8c47f774c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.114019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a384e305-40e8-4b47-a1f5-ca1b0b3b091c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:02.115275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a384e305-40e8-4b47-a1f5-ca1b0b3b091c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:02.115396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ab81a8b8-2c97-4b96-81bf-d3e8c47f774c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:02.116507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ab81a8b8-2c97-4b96-81bf-d3e8c47f774c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:02.123152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 781/1558] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:02.127618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 778/1559] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:02.127784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a012bc5-c5c4-4c93-b352-5188ab6fa6d1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.127784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a012bc5-c5c4-4c93-b352-5188ab6fa6d1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.127986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab81a8b8-2c97-4b96-81bf-d3e8c47f774c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.128089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a012bc5-c5c4-4c93-b352-5188ab6fa6d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.136083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab81a8b8-2c97-4b96-81bf-d3e8c47f774c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.140992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2384a5-dec7-4150-bbd2-101698a1cd98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.141639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2384a5-dec7-4150-bbd2-101698a1cd98 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.141922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2384a5-dec7-4150-bbd2-101698a1cd98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.142193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2384a5-dec7-4150-bbd2-101698a1cd98 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.142908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2384a5-dec7-4150-bbd2-101698a1cd98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.144203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-4a012bc5-c5c4-4c93-b352-5188ab6fa6d1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:02.144552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2384a5-dec7-4150-bbd2-101698a1cd98 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.145919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4a012bc5-c5c4-4c93-b352-5188ab6fa6d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:02.153620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 782/1560] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:02.154438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a012bc5-c5c4-4c93-b352-5188ab6fa6d1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.154914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 779/1561] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:02.156169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec222901-ae8a-47c9-95bf-641c00975bdd None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.156477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec222901-ae8a-47c9-95bf-641c00975bdd None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.156752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec222901-ae8a-47c9-95bf-641c00975bdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.165208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a012bc5-c5c4-4c93-b352-5188ab6fa6d1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.169881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.170541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.170828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.171107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.171789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.173732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.176969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ec222901-ae8a-47c9-95bf-641c00975bdd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:02.178540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ec222901-ae8a-47c9-95bf-641c00975bdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:02.186883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 780/1562] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:02.187924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec222901-ae8a-47c9-95bf-641c00975bdd None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.198334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec222901-ae8a-47c9-95bf-641c00975bdd None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.202373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 783/1563] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/users => generated 289 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.202373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.206080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.206774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.207006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.207843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.208603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:02.208906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.210508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.212317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fb36cd-0ff8-434d-a885-f6b05503c7b9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.216942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.217582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.217866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.218143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.218863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:02.219435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.219562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:02.220786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.222784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 781/1564] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:02.223814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.240710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 784/1565] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:02.240710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.241015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e5085b-9456-467a-bfd7-89b0d7d4d5ea admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.241015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.241015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.241015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.241389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.241530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:02.241839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.243459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.244786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf1cb2a-c204-42ef-96e5-227d10eda5e6 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.249211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.249849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.250128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.250399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.251126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:02.251430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.253034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.256068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 782/1566] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:02.256891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.273893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 785/1567] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:02.273893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.274161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0f72e3-2b5d-40b8-9662-9e7e2a568e9a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.277305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.277960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.278245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.278524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.278815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb705c1-02ba-4591-80cb-136df575b3b7 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.279252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:02.279559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.281189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.283334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.284199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.284519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.284792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] PATH_INFO: `/v3/projects/a75ede11bffb460aa6cb57e232cfa4d4/users/eca9bd479a194bfaabb8135605b8bf26/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.287840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:02.294071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 783/1568] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:02.295081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.304997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650f033f-dd04-4c34-804f-020971b0bb5d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.307123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=a75ede11bffb460aa6cb57e232cfa4d4, user_id=eca9bd479a194bfaabb8135605b8bf26, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.309523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.309934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.310166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.310448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.310720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.311438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:02.311742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.313338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.324717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 786/1569] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:02 2022] PUT /identity/v3/projects/a75ede11bffb460aa6cb57e232cfa4d4/users/eca9bd479a194bfaabb8135605b8bf26/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:02.325535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.340938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 784/1570] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:02.340938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.346847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a5bcf9-aeb2-4faa-ba4c-b6380c397cca admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.392879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91ba5c5-aaff-4716-af5c-dc52cb2e87ed admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.398339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.399111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.399490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.399862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.400722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.402407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.403763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.404515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.404808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.405089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.405778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.406791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:10:02.406791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:10:02.406791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:10:02.407987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.412301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 787/1571] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:02.416143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.426334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9130c545-1691-42a7-ba02-3265b519c39b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.431589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.432215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.432530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.432821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.433478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.434406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 785/1572] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/projects => generated 393 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.435505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.438439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.465095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 788/1573] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.465095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.501639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce7270d-24f9-4f29-b450-fd2de8fab449 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.528575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea28c9fd-582b-43ff-a8c0-3549f97883d2 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.533208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.533862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.534146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.534419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.535511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.537110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.549320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:02.561121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8044efe2-57e0-41cd-82c6-be56ae4d6875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.561785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044efe2-57e0-41cd-82c6-be56ae4d6875 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.562068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044efe2-57e0-41cd-82c6-be56ae4d6875 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.562345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044efe2-57e0-41cd-82c6-be56ae4d6875 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.563024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8044efe2-57e0-41cd-82c6-be56ae4d6875 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.563341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 789/1574] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/projects => generated 399 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.564144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.565144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8044efe2-57e0-41cd-82c6-be56ae4d6875 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.596223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 786/1575] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/projects => generated 385 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.596223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a3cd9-1501-4014-b8ff-0ce672569590 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.596223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a3cd9-1501-4014-b8ff-0ce672569590 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.596223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a3cd9-1501-4014-b8ff-0ce672569590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.617403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:02.621433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d45a3cd9-1501-4014-b8ff-0ce672569590 None None] MFA Rules not processed for user `eca9bd479a194bfaabb8135605b8bf26`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:02.622948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d45a3cd9-1501-4014-b8ff-0ce672569590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:02.635734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978e0efd-03e1-4346-9636-584f5c7daaa3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.687668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 787/1576] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:02.688634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45a3cd9-1501-4014-b8ff-0ce672569590 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.691366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.693001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.693288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.693562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.694545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.696518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.700158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45a3cd9-1501-4014-b8ff-0ce672569590 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.704669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.705328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.705614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.705892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.706580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.708565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.726212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 790/1577] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/projects => generated 401 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.727118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.741569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 788/1578] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/projects => generated 367 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.745551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.791521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32efdfd2-b357-4630-8a28-202f6ad10fa4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.807876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c151b76-fc55-45d5-aa70-19204e215744 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.812787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.813462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.813805 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.814089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.814809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.817450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.838389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 789/1579] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/users => generated 297 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.839106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.848708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.848890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac8aced-f819-437b-95c2-7800ba772955 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.849386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.849675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.849953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.850679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.852276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.853311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.853949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.854230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.854507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.855165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.857155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.865745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 791/1580] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:02.865745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.875820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3b9c15-7ac7-48f1-877b-2f3f6d0c183b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.879051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 790/1581] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/users => generated 287 bytes in 40 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.880422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.881429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7818214-cd73-4658-a8d1-07532599338a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.882300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7818214-cd73-4658-a8d1-07532599338a admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.882846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7818214-cd73-4658-a8d1-07532599338a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.883125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7818214-cd73-4658-a8d1-07532599338a admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.883804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7818214-cd73-4658-a8d1-07532599338a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.885779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7818214-cd73-4658-a8d1-07532599338a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.889689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d7f7d-16c4-4174-923b-55eabf4810fb admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.894287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.894961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.895251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.895532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.896207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.898587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.905457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 792/1582] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/users => generated 300 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.906576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770562ff-9993-4801-8af3-d9ebf71731d2 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.906858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770562ff-9993-4801-8af3-d9ebf71731d2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.907133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770562ff-9993-4801-8af3-d9ebf71731d2 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.908365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 793/1583] 10.222.0.20 () {54 vars in 961 bytes} [Tue Jul 5 10:10:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:02.914298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770562ff-9993-4801-8af3-d9ebf71731d2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.920269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:INSERT=11 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:02.925259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:02.932431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 791/1584] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/users => generated 293 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.932431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.936591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770562ff-9993-4801-8af3-d9ebf71731d2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.941448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.942138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.942438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.942702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.942990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4727eb27-3a49-43f9-9037-6d1121ab49dd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.943387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.945601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.947936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.948625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.948916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.949198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.949885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.951790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.965310 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 794/1585] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/users => generated 301 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.966443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.975886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 792/1586] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:02 2022] POST /identity/v3/users => generated 284 bytes in 40 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:02.975886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:02.979474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63ea060-7194-4edb-8602-69b941ffea93 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.985061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.985571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.985877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.986145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:02.986883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:02.987859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b9480-5d5e-4674-80c7-61b17fc52a05 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:02.989252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:02.993692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:02.994726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:02.995044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:02.995321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] PATH_INFO: `/v3/projects/f1f4fee9c6b844a2b7a14e2afb831f88/users/a9d95652b0714a2a8427a7e99de27bee/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.000735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 795/1587] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:02 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.001555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.011118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981abad6-2145-4c6d-8e43-53a26e39d0ba admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.011613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1f4fee9c6b844a2b7a14e2afb831f88, user_id=a9d95652b0714a2a8427a7e99de27bee, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.012684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:10:03.012684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:10:03.012684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:10:03.014633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.016268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d337a9-961a-46f0-9e88-11c2724b9332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.016944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d337a9-961a-46f0-9e88-11c2724b9332 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.017231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d337a9-961a-46f0-9e88-11c2724b9332 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.017809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d337a9-961a-46f0-9e88-11c2724b9332 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.018217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d337a9-961a-46f0-9e88-11c2724b9332 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.020087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d337a9-961a-46f0-9e88-11c2724b9332 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.028302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 793/1588] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:02 2022] PUT /identity/v3/projects/f1f4fee9c6b844a2b7a14e2afb831f88/users/a9d95652b0714a2a8427a7e99de27bee/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:03.029197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.035825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 796/1589] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.037156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ebc87-3ade-4beb-9344-1a543e577c49 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.037439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ebc87-3ade-4beb-9344-1a543e577c49 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.037714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ebc87-3ade-4beb-9344-1a543e577c49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.055948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-491ebc87-3ade-4beb-9344-1a543e577c49 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:03.057207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-491ebc87-3ade-4beb-9344-1a543e577c49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:03.090664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c688794a-be46-4eea-8e05-79c99a498e08 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.095182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.095853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.096135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.096430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.097157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.099479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.109134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 794/1590] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.109996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.120205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33eee7a-a908-4563-8ab0-094628dcd6cc admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.124413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 797/1591] 10.222.0.20 () {58 vars in 1058 bytes} [Tue Jul 5 10:10:03 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:03.124715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.125355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.125475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491ebc87-3ade-4beb-9344-1a543e577c49 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.125789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.126074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.126784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.128353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.135922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491ebc87-3ade-4beb-9344-1a543e577c49 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.141374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 795/1592] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.141374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.143266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.143921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.144207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.144513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.145272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.146862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.148554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636911c4-f558-424f-8c62-04dbb8b72595 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.153002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaba52a-857d-4626-a127-1ca252b93d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.153630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaba52a-857d-4626-a127-1ca252b93d3d admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.153913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaba52a-857d-4626-a127-1ca252b93d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.154182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaba52a-857d-4626-a127-1ca252b93d3d admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.154886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaba52a-857d-4626-a127-1ca252b93d3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.156562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaba52a-857d-4626-a127-1ca252b93d3d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.159899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 798/1593] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.159899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.170373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 796/1594] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.170373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505148fe-a590-4f19-bdd1-2ffc1454a0bd None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.170373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505148fe-a590-4f19-bdd1-2ffc1454a0bd None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.170691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5442a2c8-bfe8-4c97-a443-f767dc9b352b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.170791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505148fe-a590-4f19-bdd1-2ffc1454a0bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.175059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.175704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.176602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.176882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.177635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.179229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.186985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-505148fe-a590-4f19-bdd1-2ffc1454a0bd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:03.188178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-505148fe-a590-4f19-bdd1-2ffc1454a0bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:03.189247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 799/1595] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.190039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.197743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 797/1596] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:03.199985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e99885-5850-4aff-8b8c-cd1d94ff58f4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.201607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505148fe-a590-4f19-bdd1-2ffc1454a0bd None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.204739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.205390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.205679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.205960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.206682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.208347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.211440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505148fe-a590-4f19-bdd1-2ffc1454a0bd None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.216018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.216720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.217008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.217344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.218075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.218950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 800/1597] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.219680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.219804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.234695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 798/1598] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.234695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.234947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10a2ccc-d241-4f06-8a60-d2d2f443f1eb admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.241164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.241803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.242090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.242363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.242734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fb3ca5-eddb-464b-a963-4ad37ce8ffc8 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.247262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.247899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.248183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.248500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.249244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.250833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.260550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 799/1599] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.261326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.271184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d1752d-832a-4f4d-815f-01bafe6aca84 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.275710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.276336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.276662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.276944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.277654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.279217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.289911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 800/1600] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.290695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.299918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0585f7c-66cb-48ee-9dc3-6fc66c337b3b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.304583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.304738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.305215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.305500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.305778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.306211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.306490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.308025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.314923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 801/1601] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:03.315757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.317691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 801/1602] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:03.318471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.325966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2411d8-174d-425f-b08d-47f956c21add service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.330665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.331538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.331829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.332663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] PATH_INFO: `/v3/projects/3f2199fa0bc54901aa24195a250a9e25/users/248946c73cb245f9b8e6fe73e46133cf/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.347190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f2199fa0bc54901aa24195a250a9e25, user_id=248946c73cb245f9b8e6fe73e46133cf, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.350341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.353321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:03.367527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:03.367792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 802/1603] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:03 2022] PUT /identity/v3/projects/3f2199fa0bc54901aa24195a250a9e25/users/248946c73cb245f9b8e6fe73e46133cf/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:03.368841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.382444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8240bc97-2424-4334-9b69-5ba80ef8e6b0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.432804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.433659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.434011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.434377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.441735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d6b04-1e10-43dd-a15d-6fbad1145a8a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.444120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.445762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.446557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.447440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.447722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.447997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] PATH_INFO: `/v3/projects/38a04b3007354681be79ffb9cdc8d355/users/dd8be32629574f6fbda0f0f141ee8b16/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.456117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 802/1604] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:03.456926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.465986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] RBAC: Authorizing `identity:create_grant(project_id=38a04b3007354681be79ffb9cdc8d355, user_id=dd8be32629574f6fbda0f0f141ee8b16, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.467308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112743bb-90f5-43ea-ae90-a0ff663162de service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.469293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.472759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.474060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.474378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.474656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] PATH_INFO: `/v3/projects/e20f6d0acb324a69b6545fa8299a220f/users/20245b2b3ddd4e1a9117a723d1caa53e/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.484817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 803/1605] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:03 2022] PUT /identity/v3/projects/38a04b3007354681be79ffb9cdc8d355/users/dd8be32629574f6fbda0f0f141ee8b16/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:03.491470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e20f6d0acb324a69b6545fa8299a220f, user_id=20245b2b3ddd4e1a9117a723d1caa53e, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.494583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.498411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.513301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 803/1606] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:03 2022] PUT /identity/v3/projects/e20f6d0acb324a69b6545fa8299a220f/users/20245b2b3ddd4e1a9117a723d1caa53e/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:03.514107 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.571655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b741030-ab7d-44a3-b6da-4f08fdd71ade admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.596113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8c9003-295e-4a56-912e-1ed8a147ae53 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.600925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.601763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.602042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.602313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] PATH_INFO: `/v3/projects/04d86f178af24d009b195d3f8b89d75c/users/de97378aa74b48dbb5efe095c903b0f7/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.620849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04d86f178af24d009b195d3f8b89d75c, user_id=de97378aa74b48dbb5efe095c903b0f7, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.623567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.639548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3c082c-1db7-4a23-ab9e-8837ea53b850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.640442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c082c-1db7-4a23-ab9e-8837ea53b850 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.640777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c082c-1db7-4a23-ab9e-8837ea53b850 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.641124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3c082c-1db7-4a23-ab9e-8837ea53b850 admin admin] PATH_INFO: `/v3/projects/7eccead420ae4aacb6f31dcb17a138cc/users/036cb887c7894e75ba3eb4c1a8b21c69/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.648570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 804/1607] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:03 2022] PUT /identity/v3/projects/04d86f178af24d009b195d3f8b89d75c/users/de97378aa74b48dbb5efe095c903b0f7/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:03.650711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:03.659542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c082c-1db7-4a23-ab9e-8837ea53b850 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eccead420ae4aacb6f31dcb17a138cc, user_id=036cb887c7894e75ba3eb4c1a8b21c69, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.663624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3c082c-1db7-4a23-ab9e-8837ea53b850 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.686618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 804/1608] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:03 2022] PUT /identity/v3/projects/7eccead420ae4aacb6f31dcb17a138cc/users/036cb887c7894e75ba3eb4c1a8b21c69/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:03.688418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54011995-235b-46c2-95f5-3ce838eadbf1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.688738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54011995-235b-46c2-95f5-3ce838eadbf1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.689171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54011995-235b-46c2-95f5-3ce838eadbf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.708874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-54011995-235b-46c2-95f5-3ce838eadbf1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:03.710266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-54011995-235b-46c2-95f5-3ce838eadbf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:03.714900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:03.731211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102771a-a6ae-48e9-8976-f6c5116fb485 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:03.736318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:03.779059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 805/1609] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:03.784076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00b711c-0cee-43a3-8c06-57af2fe2100b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.784474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00b711c-0cee-43a3-8c06-57af2fe2100b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.785016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00b711c-0cee-43a3-8c06-57af2fe2100b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.803301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a528928-99e8-4243-968c-9edf35b78510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:03.804210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a528928-99e8-4243-968c-9edf35b78510 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.804534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a528928-99e8-4243-968c-9edf35b78510 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.804811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a528928-99e8-4243-968c-9edf35b78510 admin admin] PATH_INFO: `/v3/projects/534ca4147149416bb8877e247a2fd6db/users/312e5151f26b420cb29515d63ad66a10/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.817097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b00b711c-0cee-43a3-8c06-57af2fe2100b None None] MFA Rules not processed for user `dd8be32629574f6fbda0f0f141ee8b16`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:03.818413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b00b711c-0cee-43a3-8c06-57af2fe2100b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:03.822243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a528928-99e8-4243-968c-9edf35b78510 admin admin] RBAC: Authorizing `identity:create_grant(project_id=534ca4147149416bb8877e247a2fd6db, user_id=312e5151f26b420cb29515d63ad66a10, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:03.825904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a528928-99e8-4243-968c-9edf35b78510 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:03.843815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 805/1610] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:03 2022] PUT /identity/v3/projects/534ca4147149416bb8877e247a2fd6db/users/312e5151f26b420cb29515d63ad66a10/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:03.855860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1191499-655d-4175-bbec-1f7965a5a42d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.855860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1191499-655d-4175-bbec-1f7965a5a42d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.855860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1191499-655d-4175-bbec-1f7965a5a42d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.896532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a1191499-655d-4175-bbec-1f7965a5a42d None None] MFA Rules not processed for user `20245b2b3ddd4e1a9117a723d1caa53e`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:03.897947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a1191499-655d-4175-bbec-1f7965a5a42d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:03.904964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 806/1611] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:03 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:03.909733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1498fe-e18e-4914-b2e4-e7b7a356f64d None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.909733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1498fe-e18e-4914-b2e4-e7b7a356f64d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.909733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1498fe-e18e-4914-b2e4-e7b7a356f64d None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.909733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 807/1612] 10.222.0.20 () {54 vars in 912 bytes} [Tue Jul 5 10:10:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:03.919894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd41d6f-ce9b-4563-aa0c-785d28d0d389 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.920227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd41d6f-ce9b-4563-aa0c-785d28d0d389 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.920551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd41d6f-ce9b-4563-aa0c-785d28d0d389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:03.960382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8bd41d6f-ce9b-4563-aa0c-785d28d0d389 None None] MFA Rules not processed for user `de97378aa74b48dbb5efe095c903b0f7`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:03.961727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8bd41d6f-ce9b-4563-aa0c-785d28d0d389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:03.993532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 806/1613] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:03 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:03.994872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66f66a2-26e5-456a-ad67-a3e45b2462f8 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:03.995161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66f66a2-26e5-456a-ad67-a3e45b2462f8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:03.995433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66f66a2-26e5-456a-ad67-a3e45b2462f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.013751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d66f66a2-26e5-456a-ad67-a3e45b2462f8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:04.014955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d66f66a2-26e5-456a-ad67-a3e45b2462f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:04.040232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 808/1614] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:03 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:04.041645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd41d6f-ce9b-4563-aa0c-785d28d0d389 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.079341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:04.081397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 807/1615] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:04.083806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40865bba-47dc-40c8-b67b-2269f6982535 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.084205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40865bba-47dc-40c8-b67b-2269f6982535 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.084611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40865bba-47dc-40c8-b67b-2269f6982535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.111119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd41d6f-ce9b-4563-aa0c-785d28d0d389 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.114294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-40865bba-47dc-40c8-b67b-2269f6982535 None None] MFA Rules not processed for user `312e5151f26b420cb29515d63ad66a10`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:04.114672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc26b459-173a-4177-ab50-c4ce1fd4a7b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.115324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc26b459-173a-4177-ab50-c4ce1fd4a7b0 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.115603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc26b459-173a-4177-ab50-c4ce1fd4a7b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.115873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc26b459-173a-4177-ab50-c4ce1fd4a7b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.115969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-40865bba-47dc-40c8-b67b-2269f6982535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:04.125527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:04.125863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc26b459-173a-4177-ab50-c4ce1fd4a7b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.127586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc26b459-173a-4177-ab50-c4ce1fd4a7b0 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.138025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 809/1616] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:04.139405 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fa8c31-aaee-454e-8d24-56e54ccf1bb7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.139691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fa8c31-aaee-454e-8d24-56e54ccf1bb7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.139960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fa8c31-aaee-454e-8d24-56e54ccf1bb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.159012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-70fa8c31-aaee-454e-8d24-56e54ccf1bb7 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:04.160324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-70fa8c31-aaee-454e-8d24-56e54ccf1bb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:04.194198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 808/1617] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:04 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:04.196192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1738a0-104d-4271-a51b-966799aff913 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.196513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1738a0-104d-4271-a51b-966799aff913 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.196797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1738a0-104d-4271-a51b-966799aff913 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.198515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 809/1618] 10.222.0.20 () {54 vars in 961 bytes} [Tue Jul 5 10:10:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:04.199093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1738a0-104d-4271-a51b-966799aff913 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.215559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1738a0-104d-4271-a51b-966799aff913 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.220652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.221368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.221655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.221934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.222627 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.225194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.246754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 810/1619] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:10:04 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:04.247499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fa8c31-aaee-454e-8d24-56e54ccf1bb7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.254933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 810/1620] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:04 2022] POST /identity/v3/projects => generated 399 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:04.255963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.261029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fa8c31-aaee-454e-8d24-56e54ccf1bb7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.317470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371b98a7-a351-4831-b4a0-d3b1f9fd3dd5 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.321472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.322127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.322410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.322684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.333652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553317ed-ab67-4d6b-9711-7e28d06a85fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.334488 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553317ed-ab67-4d6b-9711-7e28d06a85fa admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.335122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553317ed-ab67-4d6b-9711-7e28d06a85fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.336355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553317ed-ab67-4d6b-9711-7e28d06a85fa admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.337387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553317ed-ab67-4d6b-9711-7e28d06a85fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.339605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553317ed-ab67-4d6b-9711-7e28d06a85fa admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.377654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 811/1621] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:04 2022] POST /identity/v3/projects => generated 403 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:04.379137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a74930c-1154-47b0-b4f1-5f185b84aea9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.379525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a74930c-1154-47b0-b4f1-5f185b84aea9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.380315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a74930c-1154-47b0-b4f1-5f185b84aea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.394172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.396923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.402818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-4a74930c-1154-47b0-b4f1-5f185b84aea9 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:04.404451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4a74930c-1154-47b0-b4f1-5f185b84aea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:04.408838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 811/1622] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:04.409709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.469151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:04.474660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:04.491636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1764f6c1-996d-41bb-888d-cad7c08891ee service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.505812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 812/1623] 10.222.0.20 () {58 vars in 1058 bytes} [Tue Jul 5 10:10:04 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:04.506848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a74930c-1154-47b0-b4f1-5f185b84aea9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.569692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.576593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.576593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.576593 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.603796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a74930c-1154-47b0-b4f1-5f185b84aea9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.609950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.611084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.611839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.612338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.613574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.615914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.643770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 813/1624] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:04 2022] POST /identity/v3/users => generated 306 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:04.645931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.656747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.658193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd0aa30-d811-47ed-beaa-53e9b11c35b2 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.658729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.662829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.664160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.664878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.665453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.678871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.680496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.681628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 812/1625] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:04 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:04.683900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.708497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 814/1626] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:04 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:04.711998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7952d8-2ea1-462b-88ec-25cc8d4ef143 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.712483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.725060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.727404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.727856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.728911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.729822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.733085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4f6ae4-cbe4-45f1-ab74-d1d92cb24e18 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.733863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.751530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.752319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.752982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.753336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.784981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 813/1627] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:04 2022] POST /identity/v3/users => generated 308 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:04.786542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.811065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeed678-2385-4a53-9a4b-bde1e748fa9b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.823624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.824517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.824984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.825357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.826306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.828001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.852459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 814/1628] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:04 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:04.852459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.876730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdfb5b7-2dde-4042-b0a1-7462f93a883b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.889406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.890627 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.890752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.891113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:04.892169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.894429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.902791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:04.906671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:04.935122 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 815/1629] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:04 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:04.935743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.942559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 815/1630] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:04 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:04.945580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:04.975222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016ba0c5-87ea-4e8f-9918-07c003ebc373 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.990700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc506d6-28e3-4591-a6f9-fd1cf4b425f4 service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:04.992415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:04.993384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:04.993611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:04.993883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] PATH_INFO: `/v3/projects/add7c342d3eb420cbc67880f0a50ad57/users/38f3ed9cc77540edba547a58b84b8bda/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.006052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:05.007328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.007848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.008734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] PATH_INFO: `/v3/projects/9083611e0754468bbc60631ea389c453/users/cdfdfc07221348e38c78ad53ece409a1/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.034820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=add7c342d3eb420cbc67880f0a50ad57, user_id=38f3ed9cc77540edba547a58b84b8bda, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:05.037971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:05.052430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=9083611e0754468bbc60631ea389c453, user_id=cdfdfc07221348e38c78ad53ece409a1, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:05.059613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:05.064712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 816/1631] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:04 2022] PUT /identity/v3/projects/add7c342d3eb420cbc67880f0a50ad57/users/38f3ed9cc77540edba547a58b84b8bda/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:05.066609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:05.096993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 816/1632] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:04 2022] PUT /identity/v3/projects/9083611e0754468bbc60631ea389c453/users/cdfdfc07221348e38c78ad53ece409a1/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:05.102343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:05.177129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:05.207364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:05.230868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21b2779-94ce-43eb-a051-5e4e91ba43ca admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:05.273505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e85ab3-315b-42a8-8cda-87adb86006cb admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:05.279546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:05.280572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.281305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.281305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.282254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:05.284316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:05.313913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 817/1633] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:05 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:05.320576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:05.359706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32b52e5-1888-4844-953e-1461f7499bf8 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:05.378658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ac8491-d250-4432-ac90-cd5fd929f34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:05.382449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac8491-d250-4432-ac90-cd5fd929f34c admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.383886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac8491-d250-4432-ac90-cd5fd929f34c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.384284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac8491-d250-4432-ac90-cd5fd929f34c admin admin] PATH_INFO: `/v3/projects/9083611e0754468bbc60631ea389c453/users/cdfdfc07221348e38c78ad53ece409a1/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.391027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:05.391728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.392110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.392439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.393155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:05.393786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ac8491-d250-4432-ac90-cd5fd929f34c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9083611e0754468bbc60631ea389c453, user_id=cdfdfc07221348e38c78ad53ece409a1, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:05.394804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:05.403240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ac8491-d250-4432-ac90-cd5fd929f34c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:05.421564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 817/1634] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:05 2022] GET /identity/v3/roles => generated 1420 bytes in 353 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:05.422729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:05.439320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c016669-f344-4896-9612-0f5eddf1e2b9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:05.446045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 818/1635] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:05 2022] PUT /identity/v3/projects/9083611e0754468bbc60631ea389c453/users/cdfdfc07221348e38c78ad53ece409a1/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:05.447931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fb9e4-3a61-4ac1-97cd-a55cf15c6ec5 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.448590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fb9e4-3a61-4ac1-97cd-a55cf15c6ec5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.449366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fb9e4-3a61-4ac1-97cd-a55cf15c6ec5 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.456552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 819/1636] 10.222.0.20 () {54 vars in 912 bytes} [Tue Jul 5 10:10:05 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:05.483526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474696b4-013a-402e-9f5e-e8ca3fe6810a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.483526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474696b4-013a-402e-9f5e-e8ca3fe6810a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.483526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474696b4-013a-402e-9f5e-e8ca3fe6810a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.523452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781ed4e7-000a-4504-b848-8471c3b29985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:05.524436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781ed4e7-000a-4504-b848-8471c3b29985 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.524759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781ed4e7-000a-4504-b848-8471c3b29985 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.525040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781ed4e7-000a-4504-b848-8471c3b29985 admin admin] PATH_INFO: `/v3/projects/add7c342d3eb420cbc67880f0a50ad57/users/38f3ed9cc77540edba547a58b84b8bda/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.527119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781ed4e7-000a-4504-b848-8471c3b29985 admin admin] RBAC: Authorizing `identity:create_grant(project_id=add7c342d3eb420cbc67880f0a50ad57, user_id=38f3ed9cc77540edba547a58b84b8bda, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:05.529935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781ed4e7-000a-4504-b848-8471c3b29985 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:05.537342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-474696b4-013a-402e-9f5e-e8ca3fe6810a None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:05.538702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-474696b4-013a-402e-9f5e-e8ca3fe6810a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:05.553611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 818/1637] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:05 2022] PUT /identity/v3/projects/add7c342d3eb420cbc67880f0a50ad57/users/38f3ed9cc77540edba547a58b84b8bda/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:05.553611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8126cc-0336-46a8-8899-0b41dfb2cd94 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.553611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8126cc-0336-46a8-8899-0b41dfb2cd94 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.553611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8126cc-0336-46a8-8899-0b41dfb2cd94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.616098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9b8126cc-0336-46a8-8899-0b41dfb2cd94 None None] MFA Rules not processed for user `cdfdfc07221348e38c78ad53ece409a1`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:05.624953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9b8126cc-0336-46a8-8899-0b41dfb2cd94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:05.691175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 820/1638] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:10:05 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:05.691846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9d03c-0c47-408f-80ff-9f0d001c1b9c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9d03c-0c47-408f-80ff-9f0d001c1b9c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.694766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9d03c-0c47-408f-80ff-9f0d001c1b9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.742817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-dac9d03c-0c47-408f-80ff-9f0d001c1b9c None None] MFA Rules not processed for user `38f3ed9cc77540edba547a58b84b8bda`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:05.745029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dac9d03c-0c47-408f-80ff-9f0d001c1b9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:05.780950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 819/1639] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:05 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:05.780950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8126cc-0336-46a8-8899-0b41dfb2cd94 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:05.799410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:05.869287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8126cc-0336-46a8-8899-0b41dfb2cd94 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:05.874125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:05.874448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:05.874846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:05.875254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:05.885640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 821/1640] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:05 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:05.894558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:05.897270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:05.913220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 820/1641] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:05 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:06.599585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9d03c-0c47-408f-80ff-9f0d001c1b9c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:06.614906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac9d03c-0c47-408f-80ff-9f0d001c1b9c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:06.626578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:06.626954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:06.627245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:06.627524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:06.654534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:06.666418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:06.683119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:06.688837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 822/1642] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:06 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:06.693290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c5dbd5-efa7-4b73-bd86-c18ade135ad4 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:06.699051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:06.701757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:06.702422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:06.703121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:06.714892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:06.717230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:06.744318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 821/1643] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:06 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:06.912980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:06.933677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837ca31-8d02-49a7-94d1-988d3b144a28 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:06.940319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bd8bc9-b118-49a5-aeda-9d6f78a1e21c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:06.943879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd8bc9-b118-49a5-aeda-9d6f78a1e21c service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:06.944787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd8bc9-b118-49a5-aeda-9d6f78a1e21c service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:06.945226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bd8bc9-b118-49a5-aeda-9d6f78a1e21c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:06.987954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:07.150778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bd8bc9-b118-49a5-aeda-9d6f78a1e21c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:07.154065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bd8bc9-b118-49a5-aeda-9d6f78a1e21c service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:07.185798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 823/1644] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:06 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:07.309477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:07.500043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac50d674-ff7d-4ad4-bcb5-fb489f4eedfc service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:07.508868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:07.510322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:07.510934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:07.511487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] PATH_INFO: `/v3/users/de97378aa74b48dbb5efe095c903b0f7` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:07.513139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] RBAC: Authorizing `identity:delete_user(user_id=de97378aa74b48dbb5efe095c903b0f7)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:07.514453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: WARNING py.warnings [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:10:07.514453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: warnings.warn(msg) Jul 05 10:10:07.514453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: Jul 05 10:10:07.516624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:07.726232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de97378aa74b48dbb5efe095c903b0f7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:07.728487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de97378aa74b48dbb5efe095c903b0f7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:07.772378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de97378aa74b48dbb5efe095c903b0f7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:07.775771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de97378aa74b48dbb5efe095c903b0f7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:07.802927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de97378aa74b48dbb5efe095c903b0f7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:07.922183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 822/1645] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:07 2022] DELETE /identity/v3/users/de97378aa74b48dbb5efe095c903b0f7 => generated 0 bytes in 613 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:07.927741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:07.928667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:07.929118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:07.980546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:07.982267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:08.179554 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 824/1646] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:08.238001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:08.241456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:08.310170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:08.523464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610aeba-1d75-4788-8a23-5769b6213daf admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:08.549724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b091d-ad6a-4bcf-bb3c-17a9a63cae5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:08.556349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b091d-ad6a-4bcf-bb3c-17a9a63cae5b service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:08.557239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b091d-ad6a-4bcf-bb3c-17a9a63cae5b service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:08.557239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b091d-ad6a-4bcf-bb3c-17a9a63cae5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:08.602747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8b091d-ad6a-4bcf-bb3c-17a9a63cae5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:08.605252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8b091d-ad6a-4bcf-bb3c-17a9a63cae5b service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:08.629688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 823/1647] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:10.310538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:10.312099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:10.335087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20a2720-d078-4967-9b9e-1f470b16dc26 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:10.353797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:10.354813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:10.358493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:10.359518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] PATH_INFO: `/v3/users/dd8be32629574f6fbda0f0f141ee8b16` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:10.364098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd8be32629574f6fbda0f0f141ee8b16)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:10.368266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:10.489327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd8be32629574f6fbda0f0f141ee8b16'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:10.510930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd8be32629574f6fbda0f0f141ee8b16'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:10.512528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd8be32629574f6fbda0f0f141ee8b16'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:10.513749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd8be32629574f6fbda0f0f141ee8b16'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:10.548587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd8be32629574f6fbda0f0f141ee8b16'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:10.575114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 825/1648] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:10 2022] DELETE /identity/v3/users/dd8be32629574f6fbda0f0f141ee8b16 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:10.579839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:10.581855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:10.583087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:10.631887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:10.634260 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:10.791020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 824/1649] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:10.804036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:10.805702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:10.832549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f63e2c-a3d0-4734-a8ef-dc147034dd50 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:10.850519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:10.851424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:10.851714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:10.851997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] PATH_INFO: `/v3/projects/04d86f178af24d009b195d3f8b89d75c` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:10.861495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04d86f178af24d009b195d3f8b89d75c)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:10.862777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: WARNING py.warnings [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 05 10:10:10.862777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: warnings.warn(msg) Jul 05 10:10:10.862777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: Jul 05 10:10:10.866092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:10.896843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:10.899390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:10.966589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:11.012664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04d86f178af24d009b195d3f8b89d75c', 'request_id': 'req-70c0a58c-e1b2-4bc6-8d80-510705f7b920'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.029161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04d86f178af24d009b195d3f8b89d75c', 'request_id': 'req-70c0a58c-e1b2-4bc6-8d80-510705f7b920'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.040618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0574806e-f737-444b-9c8c-0c2ed88fd19b None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:11.078734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04d86f178af24d009b195d3f8b89d75c', 'request_id': 'req-70c0a58c-e1b2-4bc6-8d80-510705f7b920'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.128229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] The token cache is being invalidate because project 04d86f178af24d009b195d3f8b89d75c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:11.128579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.136705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.142014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 826/1650] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:10 2022] DELETE /identity/v3/projects/04d86f178af24d009b195d3f8b89d75c => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:11.144667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:11.151584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:11.220609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:11.221185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:11.221480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:11.221764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:11.225315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:11.348272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:11.376258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c0a58c-e1b2-4bc6-8d80-510705f7b920 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:11.386857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:11.387917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:11.388212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:11.388521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] PATH_INFO: `/v3/users/cdfdfc07221348e38c78ad53ece409a1` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:11.390674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cdfdfc07221348e38c78ad53ece409a1)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:11.396307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:11.459189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:11.460840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:11.486194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 825/1651] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 590 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:11.489593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cdfdfc07221348e38c78ad53ece409a1'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.499657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cdfdfc07221348e38c78ad53ece409a1'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.500569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cdfdfc07221348e38c78ad53ece409a1'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.501288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cdfdfc07221348e38c78ad53ece409a1'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.519088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e764a10-8a8a-4d22-a460-ffe8aeae2559 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cdfdfc07221348e38c78ad53ece409a1'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:11.539962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 827/1652] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:11 2022] DELETE /identity/v3/users/cdfdfc07221348e38c78ad53ece409a1 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:12.755756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:12.757286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:12.788682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e609f0-41c7-4c68-aa3e-261e71e736e6 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:13.005015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.005513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.005601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.025227 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:13.026386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.026680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.027404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] PATH_INFO: `/v3/projects/38a04b3007354681be79ffb9cdc8d355` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.029052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] RBAC: Authorizing `identity:delete_project(project_id=38a04b3007354681be79ffb9cdc8d355)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:13.031120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:13.056372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] MFA Rules not processed for user `a9d95652b0714a2a8427a7e99de27bee`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:13.058058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.099905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:13.185661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 828/1653] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:10:13.186663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:13.189408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.238394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38a04b3007354681be79ffb9cdc8d355', 'request_id': 'req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.241440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9506930b-1eff-4ea8-a71e-b17e4b96501c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:13.248207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38a04b3007354681be79ffb9cdc8d355', 'request_id': 'req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.258956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38a04b3007354681be79ffb9cdc8d355', 'request_id': 'req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.273834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] The token cache is being invalidate because project 38a04b3007354681be79ffb9cdc8d355 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:13.274219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.275042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e83aa9f5-368e-4f7e-a28f-9cbde55a3cae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.281000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 826/1654] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:12 2022] DELETE /identity/v3/projects/38a04b3007354681be79ffb9cdc8d355 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:13.286556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94602d9-b960-49a0-8290-b61b2b4fd30f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.286876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94602d9-b960-49a0-8290-b61b2b4fd30f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.287347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94602d9-b960-49a0-8290-b61b2b4fd30f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.311034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a94602d9-b960-49a0-8290-b61b2b4fd30f None None] MFA Rules not processed for user `a9d95652b0714a2a8427a7e99de27bee`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:13.312590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a94602d9-b960-49a0-8290-b61b2b4fd30f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.373589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:13.382558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:13.382558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.383002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.383226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] PATH_INFO: `/v3/projects/9083611e0754468bbc60631ea389c453` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.389101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9083611e0754468bbc60631ea389c453)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:13.391070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:13.409960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 827/1655] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:13.412602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.413112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.413638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.446033 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:13.448355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.460322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 828/1656] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:13.461701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:13.463445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.495693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9083611e0754468bbc60631ea389c453', 'request_id': 'req-968adeef-590f-4ff3-ae01-37019762db7c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.506908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9083611e0754468bbc60631ea389c453', 'request_id': 'req-968adeef-590f-4ff3-ae01-37019762db7c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.521482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9083611e0754468bbc60631ea389c453', 'request_id': 'req-968adeef-590f-4ff3-ae01-37019762db7c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.545132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] The token cache is being invalidate because project 9083611e0754468bbc60631ea389c453 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:13.545132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.545531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-968adeef-590f-4ff3-ae01-37019762db7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:13.549555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 829/1657] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:13 2022] DELETE /identity/v3/projects/9083611e0754468bbc60631ea389c453 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:13.550903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c1dea-b7fd-44f3-b84e-696a0d61f769 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.551195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c1dea-b7fd-44f3-b84e-696a0d61f769 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.552043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c1dea-b7fd-44f3-b84e-696a0d61f769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.585404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c3f37-efba-4f64-8135-0caa200e64d0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:13.590516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-416c1dea-b7fd-44f3-b84e-696a0d61f769 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:13.591688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-416c1dea-b7fd-44f3-b84e-696a0d61f769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.691723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:13.709914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edee3ca5-f353-402c-b3d3-204068b1ef76 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:13.710861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee3ca5-f353-402c-b3d3-204068b1ef76 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.711147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee3ca5-f353-402c-b3d3-204068b1ef76 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.711424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee3ca5-f353-402c-b3d3-204068b1ef76 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.713790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-edee3ca5-f353-402c-b3d3-204068b1ef76 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.721910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 830/1658] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:13.723519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f8b7f-d350-439f-a949-647c300170cf None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.723631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f8b7f-d350-439f-a949-647c300170cf None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.723871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f8b7f-d350-439f-a949-647c300170cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.761500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1c4f8b7f-d350-439f-a949-647c300170cf None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:13.762731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1c4f8b7f-d350-439f-a949-647c300170cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.782944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 831/1659] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:13.783867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f8b7f-d350-439f-a949-647c300170cf None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:13.806709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f8b7f-d350-439f-a949-647c300170cf None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:13.820154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:13.820893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.821180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.821390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edee3ca5-f353-402c-b3d3-204068b1ef76 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:13.821844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.822188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:13.822498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:13.824144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:13.825472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edee3ca5-f353-402c-b3d3-204068b1ef76 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:13.839230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 829/1660] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:13 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:13.841207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c658ed-abfc-40fa-9af4-0bea346cf145 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.841752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c658ed-abfc-40fa-9af4-0bea346cf145 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.842191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c658ed-abfc-40fa-9af4-0bea346cf145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.874537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 832/1661] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:13.879143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f5c658ed-abfc-40fa-9af4-0bea346cf145 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:13.879300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:13.881324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f5c658ed-abfc-40fa-9af4-0bea346cf145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:13.901819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fc4d5a-3621-44b7-b6d0-44da23265ff8 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:13.908204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 830/1662] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:13.909325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c658ed-abfc-40fa-9af4-0bea346cf145 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:13.922753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:13.923444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.923733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.924021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.924887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:13.927409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:13.932872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c658ed-abfc-40fa-9af4-0bea346cf145 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:13.950182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:13.950899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:13.951185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:13.951476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:13.954930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:13.954930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:13.956284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:13.983842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 831/1663] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:10:13.992971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.019846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 833/1664] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/projects => generated 371 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:14.019846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.020306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e77ce6-f8f1-40a1-ad2f-fd163b72ebdc admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.025832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:14.153996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111a050a-91be-4bc5-81ef-fbf7f6f1d4ce admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.161857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.163674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.163908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.164191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.164973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.168844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.190594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.191438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.192406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.192406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.193159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.196727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.220039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 834/1665] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:14 2022] POST /identity/v3/users => generated 286 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:14.221118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.260154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a391f-7c45-4553-9135-c6299c54de31 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.261673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 832/1666] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:13 2022] POST /identity/v3/projects => generated 403 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:14.265952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.423648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2df985-b782-4030-be2a-c032c1fa1aa4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.425783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.427565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.430522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.430926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.431785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.437377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.440564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.441431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.441854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.442323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.443180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.445615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.465127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 835/1667] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:14.469159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.486725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 833/1668] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:14 2022] POST /identity/v3/users => generated 302 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:14.491639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.493715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37fb4a3-d367-44d3-b47e-c0d1d4f1e42a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.499038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.499953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.500320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.500688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.501561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.503281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.507106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4292e789-1953-413e-bc59-0cb9efdfb6f2 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.519948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 836/1669] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:14.521687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.525454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.526327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.526811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.527263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.528700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.530549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.537564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ed5737-51b0-4d79-8ec6-27ef5ef17437 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.542580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13c9ac8-79a1-4b22-8758-d37659cb56c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.543565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13c9ac8-79a1-4b22-8758-d37659cb56c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.543858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13c9ac8-79a1-4b22-8758-d37659cb56c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.544139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13c9ac8-79a1-4b22-8758-d37659cb56c7 admin admin] PATH_INFO: `/v3/projects/75cf728448e34dc3a6c4714a0b793caf/users/e18b0b5e0e244b4e94d7f58c3c9127b1/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.556132 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 834/1670] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:10:14.559764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.565953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13c9ac8-79a1-4b22-8758-d37659cb56c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75cf728448e34dc3a6c4714a0b793caf, user_id=e18b0b5e0e244b4e94d7f58c3c9127b1, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.569399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13c9ac8-79a1-4b22-8758-d37659cb56c7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.583510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:14.587620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e21f6-36e8-4719-9387-dee07414cb53 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.604243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 837/1671] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:14 2022] PUT /identity/v3/projects/75cf728448e34dc3a6c4714a0b793caf/users/e18b0b5e0e244b4e94d7f58c3c9127b1/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:14.606928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4edf1b1-f0a8-41ec-af10-413516c5a2ad None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.607228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4edf1b1-f0a8-41ec-af10-413516c5a2ad None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.607657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4edf1b1-f0a8-41ec-af10-413516c5a2ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.639617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f4edf1b1-f0a8-41ec-af10-413516c5a2ad None None] MFA Rules not processed for user `e18b0b5e0e244b4e94d7f58c3c9127b1`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:14.641084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f4edf1b1-f0a8-41ec-af10-413516c5a2ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:14.746701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.747443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.747674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.748151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.748939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.750562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.780538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 835/1672] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:14 2022] GET /identity/v3/roles => generated 1420 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:14.782135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.793434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 838/1673] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:14 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:14.794403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4edf1b1-f0a8-41ec-af10-413516c5a2ad None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.801044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50eefc-598a-4adb-85d0-3d64f6145af7 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.822894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d1f0f8-481e-4319-a3e2-df0645d7b149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.823827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d1f0f8-481e-4319-a3e2-df0645d7b149 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.824108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d1f0f8-481e-4319-a3e2-df0645d7b149 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.824380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d1f0f8-481e-4319-a3e2-df0645d7b149 admin admin] PATH_INFO: `/v3/projects/876a37cc5c744c38a6c1da63aca578ab/users/399d3739d12947428e45ca64b3b71dea/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.833229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:14.833544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4edf1b1-f0a8-41ec-af10-413516c5a2ad None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:14.845033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:14.845724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.846011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.846294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.846996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.855840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.867102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d1f0f8-481e-4319-a3e2-df0645d7b149 admin admin] RBAC: Authorizing `identity:create_grant(project_id=876a37cc5c744c38a6c1da63aca578ab, user_id=399d3739d12947428e45ca64b3b71dea, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:14.869971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d1f0f8-481e-4319-a3e2-df0645d7b149 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:14.892817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 839/1674] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:14 2022] POST /identity/v3/projects => generated 373 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:14.893864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:14.896968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 836/1675] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:14 2022] PUT /identity/v3/projects/876a37cc5c744c38a6c1da63aca578ab/users/399d3739d12947428e45ca64b3b71dea/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:14.902254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b203e-69c9-448d-95c5-baa68958e4ff None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:14.902542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b203e-69c9-448d-95c5-baa68958e4ff None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:14.903782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b203e-69c9-448d-95c5-baa68958e4ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:14.961216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-283b203e-69c9-448d-95c5-baa68958e4ff None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:14.962475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-283b203e-69c9-448d-95c5-baa68958e4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:14.974856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-162ee940-a46c-4c46-88af-44b21c7cee49 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.064226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.068327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.068672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.069020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.069949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:15.076753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:15.094474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 837/1676] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:15.103542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174f50ef-a4f6-4ec3-9a25-a78739dea391 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.103542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174f50ef-a4f6-4ec3-9a25-a78739dea391 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.103542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174f50ef-a4f6-4ec3-9a25-a78739dea391 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.103542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 838/1677] 10.222.0.20 () {54 vars in 912 bytes} [Tue Jul 5 10:10:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:15.104432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8928e54-7904-4781-9074-375ebb435d28 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.104432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8928e54-7904-4781-9074-375ebb435d28 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.104432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8928e54-7904-4781-9074-375ebb435d28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.122289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 840/1678] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:14 2022] POST /identity/v3/users => generated 293 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:15.126576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:15.136337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:15.146540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:15.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d8928e54-7904-4781-9074-375ebb435d28 None None] MFA Rules not processed for user `36ce9bd8ce6448b8b09fe9eb7b8f7c54`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:15.175898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d8928e54-7904-4781-9074-375ebb435d28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:15.257086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc6d7ab-e376-406e-9883-da0377884d22 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.257286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 839/1679] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:10:15 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:15.262655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.263338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.263624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.263903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.264358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8928e54-7904-4781-9074-375ebb435d28 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:15.287031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8928e54-7904-4781-9074-375ebb435d28 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.292005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:15.293583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:15.294834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.295782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.296305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.296958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.298173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:15.302205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:15.314350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 840/1680] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:15 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:15.315820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:15.329905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 841/1681] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:15.330887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:15.335808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d9395-659e-4fd8-8126-8d9e3de569b6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.340488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.341050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7971ea83-63c4-4609-9eca-d7a4917a58d5 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.341406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.341944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.342326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.346333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.347237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.347519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.347919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] PATH_INFO: `/v3/projects/c17673bc04044403a24371ae9097a572/users/10da914f18ca4897aa36efb2d19e0898/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.354249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:15.356723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:15.368509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 841/1682] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:15 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:15.375040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c17673bc04044403a24371ae9097a572, user_id=10da914f18ca4897aa36efb2d19e0898, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:15.378240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:15.408448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 842/1683] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:15 2022] PUT /identity/v3/projects/c17673bc04044403a24371ae9097a572/users/10da914f18ca4897aa36efb2d19e0898/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:15.410024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:15.427234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e7ddb-f597-441f-9446-40a65de41cce service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.538816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:15.549916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb614fbb-7f65-4c5a-bf80-c31ff0fdd93b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.550647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb614fbb-7f65-4c5a-bf80-c31ff0fdd93b admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.551157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb614fbb-7f65-4c5a-bf80-c31ff0fdd93b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.552232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb614fbb-7f65-4c5a-bf80-c31ff0fdd93b admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.553929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb614fbb-7f65-4c5a-bf80-c31ff0fdd93b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:15.556713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb614fbb-7f65-4c5a-bf80-c31ff0fdd93b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:15.575846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 842/1684] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:15 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:15.579017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:15.592289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1014df-931d-4a48-a06e-460a692a1989 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.597955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.599574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.599691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.599930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] PATH_INFO: `/v3/projects/c17673bc04044403a24371ae9097a572/users/10da914f18ca4897aa36efb2d19e0898/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.602479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c17673bc04044403a24371ae9097a572, user_id=10da914f18ca4897aa36efb2d19e0898, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:15.605445 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:15.635283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.635283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.635534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:15.638097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 843/1685] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:15 2022] PUT /identity/v3/projects/c17673bc04044403a24371ae9097a572/users/10da914f18ca4897aa36efb2d19e0898/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:15.674994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] MFA Rules not processed for user `10da914f18ca4897aa36efb2d19e0898`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:15.677475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:15.813528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 843/1686] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:15 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:15.827179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:15.829342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:15.965032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da0c56a-051c-4598-bfab-744f2f630af0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:15.969828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb8d2cd-68dc-4c15-ae43-b918ad832ccf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:15.970636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8d2cd-68dc-4c15-ae43-b918ad832ccf service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:15.971067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8d2cd-68dc-4c15-ae43-b918ad832ccf service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:15.971396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8d2cd-68dc-4c15-ae43-b918ad832ccf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:16.007506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb8d2cd-68dc-4c15-ae43-b918ad832ccf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:16.012913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb8d2cd-68dc-4c15-ae43-b918ad832ccf service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:16.050958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 844/1687] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:15 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:16.270786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:16.274806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:16.300234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7f8e6-bc09-4b0c-b00e-b2374608fd69 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:16.306882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37ae726-2f02-4212-be57-222775d12c68 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:16.307670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37ae726-2f02-4212-be57-222775d12c68 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:16.308109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37ae726-2f02-4212-be57-222775d12c68 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:16.308643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37ae726-2f02-4212-be57-222775d12c68 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:16.453467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37ae726-2f02-4212-be57-222775d12c68 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:16.456540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37ae726-2f02-4212-be57-222775d12c68 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:16.473914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 844/1688] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:16 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:16.536807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac30218-e011-49c7-8618-791b40b5671c None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:16.537256 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac30218-e011-49c7-8618-791b40b5671c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:16.537826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac30218-e011-49c7-8618-791b40b5671c None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:16.541880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 845/1689] 10.222.0.20 () {54 vars in 960 bytes} [Tue Jul 5 10:10:16 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:16.548854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46066f40-2f24-4d0d-b476-9b8b84185efa None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:16.549198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46066f40-2f24-4d0d-b476-9b8b84185efa None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:16.549587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46066f40-2f24-4d0d-b476-9b8b84185efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:16.580016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-46066f40-2f24-4d0d-b476-9b8b84185efa None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:16.582426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-46066f40-2f24-4d0d-b476-9b8b84185efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:16.799503 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 845/1690] 10.222.0.20 () {58 vars in 1057 bytes} [Tue Jul 5 10:10:16 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:16.810830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac30218-e011-49c7-8618-791b40b5671c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:16.847916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac30218-e011-49c7-8618-791b40b5671c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:16.858964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359a8dc0-4fbc-431d-8972-b9b2d7edfb43 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:16.859574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359a8dc0-4fbc-431d-8972-b9b2d7edfb43 service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:16.859938 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359a8dc0-4fbc-431d-8972-b9b2d7edfb43 service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:16.860127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359a8dc0-4fbc-431d-8972-b9b2d7edfb43 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:16.862057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-359a8dc0-4fbc-431d-8972-b9b2d7edfb43 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:16.891448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359a8dc0-4fbc-431d-8972-b9b2d7edfb43 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:16.893291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359a8dc0-4fbc-431d-8972-b9b2d7edfb43 service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:16.905865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 846/1691] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:16 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:10:18.982379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:18.982919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:18.983115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:19.080272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7127c0a2-434d-4bb9-b3cf-8fc5ac3832e8 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:19.081569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:19.081883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7127c0a2-434d-4bb9-b3cf-8fc5ac3832e8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:19.082422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7127c0a2-434d-4bb9-b3cf-8fc5ac3832e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:19.087786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] MFA Rules not processed for user `036cb887c7894e75ba3eb4c1a8b21c69`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:19.090722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:19.164284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-7127c0a2-434d-4bb9-b3cf-8fc5ac3832e8 None None] MFA Rules not processed for user `399d3739d12947428e45ca64b3b71dea`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:19.179811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7127c0a2-434d-4bb9-b3cf-8fc5ac3832e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:19.299598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 846/1692] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:19.303102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:19.307458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:19.367674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:19.416687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 847/1693] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:19 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:19.417737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7127c0a2-434d-4bb9-b3cf-8fc5ac3832e8 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:19.475795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10cadc3-b617-4e5e-9c14-71e09d1c6fb0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:19.486650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:19.487542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:19.487831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:19.488110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] PATH_INFO: `/v3/users/20245b2b3ddd4e1a9117a723d1caa53e` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:19.491060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] RBAC: Authorizing `identity:delete_user(user_id=20245b2b3ddd4e1a9117a723d1caa53e)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:19.494148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:19.612923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7127c0a2-434d-4bb9-b3cf-8fc5ac3832e8 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:19.637295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2968f9c8-24c5-4738-9220-9960ec117369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:19.639754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968f9c8-24c5-4738-9220-9960ec117369 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:19.640377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968f9c8-24c5-4738-9220-9960ec117369 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:19.641402 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2968f9c8-24c5-4738-9220-9960ec117369 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:19.643871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2968f9c8-24c5-4738-9220-9960ec117369 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:19.644588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20245b2b3ddd4e1a9117a723d1caa53e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:19.646323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20245b2b3ddd4e1a9117a723d1caa53e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:19.650328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2968f9c8-24c5-4738-9220-9960ec117369 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:19.672714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20245b2b3ddd4e1a9117a723d1caa53e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:19.674962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20245b2b3ddd4e1a9117a723d1caa53e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:19.699878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20245b2b3ddd4e1a9117a723d1caa53e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:19.747210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 848/1694] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:19 2022] POST /identity/v3/projects => generated 401 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:19.748725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:19.749025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:19.749310 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:19.774158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 847/1695] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:19 2022] DELETE /identity/v3/users/20245b2b3ddd4e1a9117a723d1caa53e => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:19.774158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:19.804016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:19.820723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] MFA Rules not processed for user `248946c73cb245f9b8e6fe73e46133cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:19.822140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:19.857723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:20.026654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a386a1-6b3d-4c79-9e6c-371b47fabadf admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:20.050088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 849/1696] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:20.053275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:20.061427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:20.077438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3e417e-7678-4b9b-8662-af447e7101e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:20.082925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e417e-7678-4b9b-8662-af447e7101e8 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.086567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e417e-7678-4b9b-8662-af447e7101e8 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.090095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e417e-7678-4b9b-8662-af447e7101e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.115242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8c3e417e-7678-4b9b-8662-af447e7101e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:20.272485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3e417e-7678-4b9b-8662-af447e7101e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:20.274529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3e417e-7678-4b9b-8662-af447e7101e8 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:20.304975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 848/1697] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:19 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:20.306579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f887d23d-464d-484d-94da-f707d623af38 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.306874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f887d23d-464d-484d-94da-f707d623af38 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.307157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f887d23d-464d-484d-94da-f707d623af38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.370676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f887d23d-464d-484d-94da-f707d623af38 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:20.373968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f887d23d-464d-484d-94da-f707d623af38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:20.375995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4aa4e5-c21f-46e9-b461-dd00a3cdb3df None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:20.389185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:20.410292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 849/1698] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:20.410449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.410449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.410449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.410706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78013826-86c1-4331-87d2-6049851d64cf None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.410706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78013826-86c1-4331-87d2-6049851d64cf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.411034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78013826-86c1-4331-87d2-6049851d64cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.418154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:20.423578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:20.465071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-78013826-86c1-4331-87d2-6049851d64cf None None] MFA Rules not processed for user `248946c73cb245f9b8e6fe73e46133cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:20.466239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:20.466391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-78013826-86c1-4331-87d2-6049851d64cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:20.495389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 850/1699] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:20 2022] POST /identity/v3/users => generated 307 bytes in 442 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:20.495389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:20.496880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 850/1700] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:20.497418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78013826-86c1-4331-87d2-6049851d64cf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:20.507197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:20.530014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78013826-86c1-4331-87d2-6049851d64cf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:20.541099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:20.541801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.542092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.543206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.543878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:20.545569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:20.572204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 851/1701] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:20 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:20.573129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:20.578977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:20.631562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4e69d6-16b7-4d28-836a-5443ccbd5f0c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:20.654673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:20.655365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.655655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.655934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.692841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:20.695736 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:20.718849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 852/1702] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:20.720752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:20.735667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f0100e-f3f6-4a5b-9258-24e15585a6ec service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:20.742052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37b323d-22cc-4040-afe1-69fa8fe2faf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:20.743428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37b323d-22cc-4040-afe1-69fa8fe2faf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.743971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37b323d-22cc-4040-afe1-69fa8fe2faf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.744466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37b323d-22cc-4040-afe1-69fa8fe2faf9 admin admin] PATH_INFO: `/v3/projects/f128e133c7264725bb1a89f59964e0cf/users/4b04d8ffacb34c89b2c8110009867cf6/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.781308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37b323d-22cc-4040-afe1-69fa8fe2faf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f128e133c7264725bb1a89f59964e0cf, user_id=4b04d8ffacb34c89b2c8110009867cf6, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:20.800842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37b323d-22cc-4040-afe1-69fa8fe2faf9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:20.855516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 853/1703] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:20 2022] PUT /identity/v3/projects/f128e133c7264725bb1a89f59964e0cf/users/4b04d8ffacb34c89b2c8110009867cf6/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:20.862082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5014a594-5c4c-4486-84aa-4c8b86605506 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:20.862604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64640f8-1740-487d-86dd-f73b3065ba12 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:20.868982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5014a594-5c4c-4486-84aa-4c8b86605506 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:20.869523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5014a594-5c4c-4486-84aa-4c8b86605506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:20.991755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5014a594-5c4c-4486-84aa-4c8b86605506 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:20.998207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5014a594-5c4c-4486-84aa-4c8b86605506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:21.116716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:21.126025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:21.126675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:21.127102 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:21.192251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 854/1704] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:10:20 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:21.195730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5014a594-5c4c-4486-84aa-4c8b86605506 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:21.210326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:21.345762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:21.347526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:21.378775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 851/1705] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 881 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:21.409240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5014a594-5c4c-4486-84aa-4c8b86605506 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:21.417868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:21.419571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:21.419755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:21.420040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:21.420891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:21.423818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:21.446202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 855/1706] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:21 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:21.446627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:21.488469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028b7864-fa8d-4495-98dc-0b00aa1cb6d4 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:21.512533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:21.514961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:21.525203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:21.527870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:21.528541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:21.529824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] PATH_INFO: `/v3/projects/f128e133c7264725bb1a89f59964e0cf/users/4b04d8ffacb34c89b2c8110009867cf6/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:21.533566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f128e133c7264725bb1a89f59964e0cf, user_id=4b04d8ffacb34c89b2c8110009867cf6, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:21.538486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:21.603067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 852/1707] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:21 2022] PUT /identity/v3/projects/f128e133c7264725bb1a89f59964e0cf/users/4b04d8ffacb34c89b2c8110009867cf6/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:21.686660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:21.688770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:21.722839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ba8bf4-9978-4042-85ec-e91d99810e63 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:21.801161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:21.836709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545272bc-bb79-4324-a4b9-29ffb39ab852 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:21.849991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:21.850981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:21.851265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:21.851540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] PATH_INFO: `/v3/users/eca9bd479a194bfaabb8135605b8bf26` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:21.853772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] RBAC: Authorizing `identity:delete_user(user_id=eca9bd479a194bfaabb8135605b8bf26)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:21.855719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:21.858989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:21.859854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:21.860152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:21.860543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:21.976029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eca9bd479a194bfaabb8135605b8bf26'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:21.994129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eca9bd479a194bfaabb8135605b8bf26'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:21.995095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eca9bd479a194bfaabb8135605b8bf26'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:21.995859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eca9bd479a194bfaabb8135605b8bf26'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.026413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-9b952035-f72b-4dbe-b0f7-5bfca495760b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eca9bd479a194bfaabb8135605b8bf26'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.027077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:22.068825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 853/1708] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:21 2022] DELETE /identity/v3/users/eca9bd479a194bfaabb8135605b8bf26 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:22.069325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:22.070975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:22.071423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:22.071703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:22.072007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:22.099557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 856/1709] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:10:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 588 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:10:22.120691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:22.121948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:22.351508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 854/1710] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:22.454248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:22.521701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:22.523146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:22.545564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a85125-a95a-483c-a4a7-3581ad4cf518 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:22.550119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cf1ad8-e563-4123-807c-9a55f6ad67df service swift] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:22.557393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:22.558452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:22.558889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:22.559308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] PATH_INFO: `/v3/projects/e20f6d0acb324a69b6545fa8299a220f` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:22.561260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e20f6d0acb324a69b6545fa8299a220f)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:22.563373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:22.637637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e20f6d0acb324a69b6545fa8299a220f', 'request_id': 'req-66e29151-4c4d-4415-8b4d-29903b93e379'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.647825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e20f6d0acb324a69b6545fa8299a220f', 'request_id': 'req-66e29151-4c4d-4415-8b4d-29903b93e379'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.651146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:22.657159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e20f6d0acb324a69b6545fa8299a220f', 'request_id': 'req-66e29151-4c4d-4415-8b4d-29903b93e379'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.680690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] The token cache is being invalidate because project e20f6d0acb324a69b6545fa8299a220f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:22.681165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.681838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.686798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 855/1711] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:22 2022] DELETE /identity/v3/projects/e20f6d0acb324a69b6545fa8299a220f => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:22.688684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:22.691024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:22.702351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712eaad7-586c-4907-94ce-ccb750309686 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:22.703779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712eaad7-586c-4907-94ce-ccb750309686 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:22.704351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712eaad7-586c-4907-94ce-ccb750309686 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:22.704845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712eaad7-586c-4907-94ce-ccb750309686 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:22.707555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-712eaad7-586c-4907-94ce-ccb750309686 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:22.725890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712eaad7-586c-4907-94ce-ccb750309686 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:22.728122 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712eaad7-586c-4907-94ce-ccb750309686 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:22.743547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 857/1712] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:22.786724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e29151-4c4d-4415-8b4d-29903b93e379 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:22.792217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:22.793143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:22.793427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:22.793703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] PATH_INFO: `/v3/users/38f3ed9cc77540edba547a58b84b8bda` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:22.795117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38f3ed9cc77540edba547a58b84b8bda)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:22.797588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:22.906292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38f3ed9cc77540edba547a58b84b8bda'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.927866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38f3ed9cc77540edba547a58b84b8bda'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.932450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38f3ed9cc77540edba547a58b84b8bda'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.934886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38f3ed9cc77540edba547a58b84b8bda'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.970350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38f3ed9cc77540edba547a58b84b8bda'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:22.997754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 856/1713] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:22 2022] DELETE /identity/v3/users/38f3ed9cc77540edba547a58b84b8bda => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:23.810380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c5938-7c37-4926-83dc-144c1776f4af None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:23.819840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c5938-7c37-4926-83dc-144c1776f4af None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:23.819840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c5938-7c37-4926-83dc-144c1776f4af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:23.857888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9e3c5938-7c37-4926-83dc-144c1776f4af None None] MFA Rules not processed for user `248946c73cb245f9b8e6fe73e46133cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:23.865290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9e3c5938-7c37-4926-83dc-144c1776f4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:24.023323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:24.024821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:24.160435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 858/1714] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:24.201662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5306ad-839d-4074-adf7-28b0a783d5e7 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:24.241023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:24.241936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:24.242225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:24.242508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] PATH_INFO: `/v3/projects/a75ede11bffb460aa6cb57e232cfa4d4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:24.245349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a75ede11bffb460aa6cb57e232cfa4d4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:24.247572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:24.285298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:24.394767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a75ede11bffb460aa6cb57e232cfa4d4', 'request_id': 'req-cd819c5c-b645-40d1-a9c3-be0863d67191'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:24.412095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a75ede11bffb460aa6cb57e232cfa4d4', 'request_id': 'req-cd819c5c-b645-40d1-a9c3-be0863d67191'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:24.431763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a75ede11bffb460aa6cb57e232cfa4d4', 'request_id': 'req-cd819c5c-b645-40d1-a9c3-be0863d67191'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:24.461372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] The token cache is being invalidate because project a75ede11bffb460aa6cb57e232cfa4d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:24.461704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:24.463634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd819c5c-b645-40d1-a9c3-be0863d67191 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:24.470529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 857/1715] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:24 2022] DELETE /identity/v3/projects/a75ede11bffb460aa6cb57e232cfa4d4 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:24.472554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36bbe9f-212c-4a06-badc-ad10e752cc25 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:24.472843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36bbe9f-212c-4a06-badc-ad10e752cc25 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:24.473392 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36bbe9f-212c-4a06-badc-ad10e752cc25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:24.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b36bbe9f-212c-4a06-badc-ad10e752cc25 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:24.565225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b36bbe9f-212c-4a06-badc-ad10e752cc25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:24.721659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:24.772339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 859/1716] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:24.780025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1b719f-d4cf-4998-89db-e6dd2a4ad4e3 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:24.780622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1b719f-d4cf-4998-89db-e6dd2a4ad4e3 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:24.781140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1b719f-d4cf-4998-89db-e6dd2a4ad4e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:24.829869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ec1b719f-d4cf-4998-89db-e6dd2a4ad4e3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:24.834681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ec1b719f-d4cf-4998-89db-e6dd2a4ad4e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:24.849751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 858/1717] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:24.853997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36bbe9f-212c-4a06-badc-ad10e752cc25 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:24.868531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36bbe9f-212c-4a06-badc-ad10e752cc25 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:24.883202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b3f010-7877-42e3-9d45-1e591005b3ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:24.884006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3f010-7877-42e3-9d45-1e591005b3ed admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:24.884410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3f010-7877-42e3-9d45-1e591005b3ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:24.901928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b3f010-7877-42e3-9d45-1e591005b3ed admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:24.903132 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b3f010-7877-42e3-9d45-1e591005b3ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:24.908887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b3f010-7877-42e3-9d45-1e591005b3ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:24.913043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b3f010-7877-42e3-9d45-1e591005b3ed admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:24.977414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 860/1718] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:24.981056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1b719f-d4cf-4998-89db-e6dd2a4ad4e3 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.002837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1b719f-d4cf-4998-89db-e6dd2a4ad4e3 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.017876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.021621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.021897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.022172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.025656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.033889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.092262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2efa83-dff0-48dd-9d7f-496162078675 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.092832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2efa83-dff0-48dd-9d7f-496162078675 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.093177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2efa83-dff0-48dd-9d7f-496162078675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.128117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 859/1719] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:24 2022] POST /identity/v3/projects => generated 373 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:25.129500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.158412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-da2efa83-dff0-48dd-9d7f-496162078675 None None] MFA Rules not processed for user `36ce9bd8ce6448b8b09fe9eb7b8f7c54`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:25.159679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-da2efa83-dff0-48dd-9d7f-496162078675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:25.257525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9bdf2a-752c-4f3a-83d9-672956b83f69 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.266446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.267820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.268557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.269092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.270051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.273749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.319888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 860/1720] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:25 2022] POST /identity/v3/users => generated 287 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:25.323683 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.326013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 861/1721] 10.222.0.30 () {58 vars in 1015 bytes} [Tue Jul 5 10:10:25 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:25.339881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efdd9b6-8f0a-45ff-a9e3-0fd22751e42e admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.347626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.349665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.350557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.351090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.353166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.360347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ee03dd-1944-4762-8a62-9fe74ddbd7c2 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.360696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ee03dd-1944-4762-8a62-9fe74ddbd7c2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.360984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ee03dd-1944-4762-8a62-9fe74ddbd7c2 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.362660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 862/1722] 10.222.0.20 () {54 vars in 912 bytes} [Tue Jul 5 10:10:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 05 10:10:25.364363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.388163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ea04ed-3e83-4cb4-9376-2f89f3f8780e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.388163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ea04ed-3e83-4cb4-9376-2f89f3f8780e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.388163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ea04ed-3e83-4cb4-9376-2f89f3f8780e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.418567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-72ea04ed-3e83-4cb4-9376-2f89f3f8780e None None] MFA Rules not processed for user `36ce9bd8ce6448b8b09fe9eb7b8f7c54`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:25.419883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-72ea04ed-3e83-4cb4-9376-2f89f3f8780e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:25.425113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 861/1723] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:25 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:25.428932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.435626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 863/1724] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:10:25 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Jul 05 10:10:25.436772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ea04ed-3e83-4cb4-9376-2f89f3f8780e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.437591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:25.461568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ea04ed-3e83-4cb4-9376-2f89f3f8780e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.473201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.473935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.474220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.474501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.475258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.484474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.502420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 864/1725] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:25 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:25.503302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.506181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:25.507491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8078ee3-532c-4055-be9e-66c58e2d3784 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.547852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242edb4-617c-402e-81c8-1af3e6a01c17 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.551132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.552085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.552319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.552467 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.553015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.553307 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.553589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.555778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] PATH_INFO: `/v3/projects/add7c342d3eb420cbc67880f0a50ad57` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.559219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] RBAC: Authorizing `identity:delete_project(project_id=add7c342d3eb420cbc67880f0a50ad57)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.561684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.575702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.577204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.596302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 865/1726] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:25 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:25.597832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.622899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f4ce61-c1d4-48da-b8b5-e650f72a4681 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.638799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.639494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.639851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.640022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.655757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.657250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.677202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 866/1727] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:25 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:25.678134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:25.701248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c749c2-cebf-4e05-b0e1-938e13198dbd service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:25.751145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=10,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:25.766744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'add7c342d3eb420cbc67880f0a50ad57', 'request_id': 'req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:25.783477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'add7c342d3eb420cbc67880f0a50ad57', 'request_id': 'req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:25.788107 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6b59af-73ba-47d8-8ee6-18e751a974d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:25.789246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b59af-73ba-47d8-8ee6-18e751a974d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.789636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b59af-73ba-47d8-8ee6-18e751a974d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.790006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b59af-73ba-47d8-8ee6-18e751a974d7 admin admin] PATH_INFO: `/v3/projects/fe78696ea79041729421265731148164/users/6512b355e1ce4824ad69eb9e23cbea95/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.796940 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'add7c342d3eb420cbc67880f0a50ad57', 'request_id': 'req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:25.839973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6b59af-73ba-47d8-8ee6-18e751a974d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe78696ea79041729421265731148164, user_id=6512b355e1ce4824ad69eb9e23cbea95, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:25.840531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] The token cache is being invalidate because project add7c342d3eb420cbc67880f0a50ad57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:25.843956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:25.850483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b6923d2c-c6e1-4e01-9dd6-4d9b2217fc9b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:25.851151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6b59af-73ba-47d8-8ee6-18e751a974d7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:25.861581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 862/1728] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:25 2022] DELETE /identity/v3/projects/add7c342d3eb420cbc67880f0a50ad57 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:25.867955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca6120-ea50-48b8-9da4-97d00360ce87 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.868234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca6120-ea50-48b8-9da4-97d00360ce87 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.869101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca6120-ea50-48b8-9da4-97d00360ce87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.897727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 867/1729] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:25 2022] PUT /identity/v3/projects/fe78696ea79041729421265731148164/users/6512b355e1ce4824ad69eb9e23cbea95/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:25.902690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae3d7a1-51c6-460f-bf0e-81e7281f1bc4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:25.903288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae3d7a1-51c6-460f-bf0e-81e7281f1bc4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:25.905217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae3d7a1-51c6-460f-bf0e-81e7281f1bc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:25.945299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fdca6120-ea50-48b8-9da4-97d00360ce87 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:25.952488 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fdca6120-ea50-48b8-9da4-97d00360ce87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:26.005999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9ae3d7a1-51c6-460f-bf0e-81e7281f1bc4 None None] MFA Rules not processed for user `6512b355e1ce4824ad69eb9e23cbea95`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:26.009115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9ae3d7a1-51c6-460f-bf0e-81e7281f1bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:26.037032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:26.136451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 868/1730] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:25 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:26.145308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102179f4-a4bb-4dcd-a381-2ecc66a22196 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:26.145749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102179f4-a4bb-4dcd-a381-2ecc66a22196 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:26.146136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102179f4-a4bb-4dcd-a381-2ecc66a22196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:26.201621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 863/1731] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:26.204493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eedc18-fdea-4c85-875f-9f1161f54f14 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:26.205176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eedc18-fdea-4c85-875f-9f1161f54f14 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:26.205974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eedc18-fdea-4c85-875f-9f1161f54f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:26.214716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-102179f4-a4bb-4dcd-a381-2ecc66a22196 None None] MFA Rules not processed for user `036cb887c7894e75ba3eb4c1a8b21c69`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:26.216044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-102179f4-a4bb-4dcd-a381-2ecc66a22196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:26.256130 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-21eedc18-fdea-4c85-875f-9f1161f54f14 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:26.258010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-21eedc18-fdea-4c85-875f-9f1161f54f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:26.282507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 864/1732] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:26.292910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21eedc18-fdea-4c85-875f-9f1161f54f14 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:26.332375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21eedc18-fdea-4c85-875f-9f1161f54f14 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:26.362973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:26.365189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:26.367404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:26.368875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:26.370950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:26.373435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:26.382169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:26.382522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 869/1733] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:26.425328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 865/1734] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:26.429905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102179f4-a4bb-4dcd-a381-2ecc66a22196 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:26.445499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102179f4-a4bb-4dcd-a381-2ecc66a22196 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:26.462215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:26.462754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:26.463111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:26.463460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:26.464174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:26.466204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:26.536792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 870/1735] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:26 2022] POST /identity/v3/projects => generated 381 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:26.542727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:26.558687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4779c0-d963-400e-b1db-2f2c5ead7437 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:26.648168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:26.649110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:26.649475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:26.649661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:26.650422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:26.652423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:26.685257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 866/1736] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:26 2022] POST /identity/v3/users => generated 291 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:26.689017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:26.741528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4b167c-76c7-46a4-9962-a6e1f6ba4645 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:26.757064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:26.759069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:26.759339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:26.769869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:26.773052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:26.775668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:26.779678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:26.796761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:26.850471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 871/1737] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:26 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:26.854470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:26.903533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed373db-f655-4dbb-812b-9df88ed63a55 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:26.936023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0de989-e0b5-4b28-9f51-5a0b3f103a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:26.940313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0de989-e0b5-4b28-9f51-5a0b3f103a7e admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:26.940584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0de989-e0b5-4b28-9f51-5a0b3f103a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:26.942137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0de989-e0b5-4b28-9f51-5a0b3f103a7e admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:26.943476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0de989-e0b5-4b28-9f51-5a0b3f103a7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:26.959887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0de989-e0b5-4b28-9f51-5a0b3f103a7e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:27.006532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:27.038692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 872/1738] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:26 2022] GET /identity/v3/roles => generated 1420 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:27.039572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:27.047262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:27.047551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:27.170659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc06f8ba-325a-41fc-b315-c5f17a714258 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:27.183930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] MFA Rules not processed for user `4b04d8ffacb34c89b2c8110009867cf6`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:27.185218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:27.190804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:27.191509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:27.191792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:27.192076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:27.456209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:27.462853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:27.475604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:27.512480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 867/1739] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:26 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 735 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:27.513452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:27.546982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 873/1740] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:27 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 506 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:27.569433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf575f7-7441-40c6-be19-04bc08aa38ca service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:27.570383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:27.576221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:27.613229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937580d1-80e7-467a-aaf6-e98fd61428bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:27.615203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937580d1-80e7-467a-aaf6-e98fd61428bd admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:27.615372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937580d1-80e7-467a-aaf6-e98fd61428bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:27.615764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937580d1-80e7-467a-aaf6-e98fd61428bd admin admin] PATH_INFO: `/v3/projects/ceb99c8505a549a2976bde811ad29d14/users/f55fd1108d15468489d5c4ba958fb1fd/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:27.670027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937580d1-80e7-467a-aaf6-e98fd61428bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=ceb99c8505a549a2976bde811ad29d14, user_id=f55fd1108d15468489d5c4ba958fb1fd, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:27.679758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a5162-6608-42ca-8887-9e120a4a05c4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:27.680895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937580d1-80e7-467a-aaf6-e98fd61428bd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:27.698431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:27.700904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:27.701733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:27.703672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] PATH_INFO: `/v3/users/312e5151f26b420cb29515d63ad66a10` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:27.707263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] RBAC: Authorizing `identity:delete_user(user_id=312e5151f26b420cb29515d63ad66a10)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:27.712679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:27.712855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 868/1741] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:27 2022] PUT /identity/v3/projects/ceb99c8505a549a2976bde811ad29d14/users/f55fd1108d15468489d5c4ba958fb1fd/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:27.725665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:27.726230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:27.726765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:27.806773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] MFA Rules not processed for user `f55fd1108d15468489d5c4ba958fb1fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:27.808987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:27.921114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '312e5151f26b420cb29515d63ad66a10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:27.924898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '312e5151f26b420cb29515d63ad66a10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:27.958717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '312e5151f26b420cb29515d63ad66a10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:27.963273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '312e5151f26b420cb29515d63ad66a10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:27.986042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-11f60823-fc2c-4155-bcc2-42999dc1a94b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '312e5151f26b420cb29515d63ad66a10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:28.052048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 869/1742] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:27 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:10:28.056282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:28.061867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:28.071856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 874/1743] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:27 2022] DELETE /identity/v3/users/312e5151f26b420cb29515d63ad66a10 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:28.074113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:28.074436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:28.074766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:28.165176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:28.166406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:28.187064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:28.253925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fc8532-6311-4f50-bfdc-c4f40cfd059b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:28.409862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 875/1744] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:10:28.418053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:28.419122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:28.419601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:28.420044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:28.421268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:28.423753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:28.425344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:28.430401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:28.490237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 870/1745] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:28 2022] POST /identity/v3/projects => generated 381 bytes in 434 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:28.494497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:28.591425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b2870-a91f-4ed1-b0fb-1acacedb6291 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:28.709343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952738ee-967b-444b-8199-630a5b18683d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:28.716954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:28.719499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:28.721331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:28.721331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:28.721331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:28.728185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:28.735791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:28.796975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 871/1746] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:28 2022] POST /identity/v3/users => generated 297 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:28.800723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:28.824137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22e1518-ed89-48b4-8d3d-c85b23d0ef2b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:28.847302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:28.848271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:28.848674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:28.849076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:28.849919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:28.851669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:28.857143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:28.857824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:28.858204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:28.858391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:28.883496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:28.885664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:28.897934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 872/1747] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:28 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:10:28.910283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:28.935395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 876/1748] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 512 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:28.936221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc3f8ce-07b9-452f-bb3e-03437604ea66 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:28.943299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:28.944343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:28.944755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:28.945138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] PATH_INFO: `/v3/projects/4103d681bda344e2ab20233b677d40fb/users/d7b549b96cab49eda16ebb026666f8d2/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:28.969552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4103d681bda344e2ab20233b677d40fb, user_id=d7b549b96cab49eda16ebb026666f8d2, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:28.969552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:28.997794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 873/1749] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:28 2022] PUT /identity/v3/projects/4103d681bda344e2ab20233b677d40fb/users/d7b549b96cab49eda16ebb026666f8d2/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:29.022556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:29.022556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d578be4-98e5-4eba-83bc-a8f09970d2a8 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:29.231638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf0bcfe-ab11-4c7e-8d7c-a7965a170a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:29.232541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0bcfe-ab11-4c7e-8d7c-a7965a170a77 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:29.232820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0bcfe-ab11-4c7e-8d7c-a7965a170a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:29.233119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0bcfe-ab11-4c7e-8d7c-a7965a170a77 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:29.233848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf0bcfe-ab11-4c7e-8d7c-a7965a170a77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:29.235489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf0bcfe-ab11-4c7e-8d7c-a7965a170a77 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:29.285292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:29.287797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 877/1750] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:29 2022] GET /identity/v3/roles => generated 1420 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:29.302046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129d2b86-17fb-40f4-ba6d-ed34e62a2cb6 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:29.316633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:29.317882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:29.318268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:29.318647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] PATH_INFO: `/v3/projects/4103d681bda344e2ab20233b677d40fb/users/d7b549b96cab49eda16ebb026666f8d2/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:29.322875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4103d681bda344e2ab20233b677d40fb, user_id=d7b549b96cab49eda16ebb026666f8d2, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:29.341029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:29.377681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 874/1751] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:29 2022] PUT /identity/v3/projects/4103d681bda344e2ab20233b677d40fb/users/d7b549b96cab49eda16ebb026666f8d2/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:29.387810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358e836d-1eda-4fe0-8b84-1cc61c230047 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:29.388105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358e836d-1eda-4fe0-8b84-1cc61c230047 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:29.388443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358e836d-1eda-4fe0-8b84-1cc61c230047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:29.481755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-358e836d-1eda-4fe0-8b84-1cc61c230047 None None] MFA Rules not processed for user `d7b549b96cab49eda16ebb026666f8d2`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:29.500798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-358e836d-1eda-4fe0-8b84-1cc61c230047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:29.842955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 878/1752] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:29 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 456 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:29.856592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:29.858796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:29.931323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:30.038603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a254e35-b7e8-4787-a150-f00744fb17a5 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:30.047642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:30.048784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:30.049288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:30.049995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:30.089066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:30.094688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:30.139372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 875/1753] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:29 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:30.297801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358e836d-1eda-4fe0-8b84-1cc61c230047 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:30.311565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358e836d-1eda-4fe0-8b84-1cc61c230047 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:30.330515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3824fc-c9b4-4ea2-ae42-c81f5d697f0e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:30.331206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3824fc-c9b4-4ea2-ae42-c81f5d697f0e service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:30.331494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3824fc-c9b4-4ea2-ae42-c81f5d697f0e service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:30.331809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3824fc-c9b4-4ea2-ae42-c81f5d697f0e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:30.411207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:30.413187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:30.439851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:30.465599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3824fc-c9b4-4ea2-ae42-c81f5d697f0e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:30.467087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3824fc-c9b4-4ea2-ae42-c81f5d697f0e service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:30.487621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 879/1754] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:30 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:30.661890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe229c6d-de4f-449a-9bb3-08dfef9130d6 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:30.678157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:30.679087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:30.679370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:30.679646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] PATH_INFO: `/v3/users/6512b355e1ce4824ad69eb9e23cbea95` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:30.681118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6512b355e1ce4824ad69eb9e23cbea95)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:30.688189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:30.848172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6512b355e1ce4824ad69eb9e23cbea95'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:30.880183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6512b355e1ce4824ad69eb9e23cbea95'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:30.883191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6512b355e1ce4824ad69eb9e23cbea95'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:30.899342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6512b355e1ce4824ad69eb9e23cbea95'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:30.970836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6512b355e1ce4824ad69eb9e23cbea95'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:31.028703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 876/1755] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:30 2022] DELETE /identity/v3/users/6512b355e1ce4824ad69eb9e23cbea95 => generated 0 bytes in 618 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:31.034663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18df0892-fc04-48f2-9db9-7f3462515d43 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:31.035302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18df0892-fc04-48f2-9db9-7f3462515d43 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:31.035654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18df0892-fc04-48f2-9db9-7f3462515d43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:31.087980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-18df0892-fc04-48f2-9db9-7f3462515d43 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:31.090462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-18df0892-fc04-48f2-9db9-7f3462515d43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:31.362612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 880/1756] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:31.528845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:31.530398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:31.667423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:31.706279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd79771-9c7f-42b4-a0fb-f3b68dd247d2 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:31.718771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d96fa-7fb2-452d-aaa7-29fdd508cd41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:31.720574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d96fa-7fb2-452d-aaa7-29fdd508cd41 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:31.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d96fa-7fb2-452d-aaa7-29fdd508cd41 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:31.721629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d96fa-7fb2-452d-aaa7-29fdd508cd41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:31.755758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631d96fa-7fb2-452d-aaa7-29fdd508cd41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:31.757445 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631d96fa-7fb2-452d-aaa7-29fdd508cd41 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:31.793950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 877/1757] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:32.125832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b8b686-2028-4720-afa6-666b3508102d None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:32.126199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b8b686-2028-4720-afa6-666b3508102d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:32.126727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b8b686-2028-4720-afa6-666b3508102d None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:32.128319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 881/1758] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:10:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:32.301129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7fe553-f253-46d0-85e2-b2645fb77df4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:32.302183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7fe553-f253-46d0-85e2-b2645fb77df4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:32.302956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7fe553-f253-46d0-85e2-b2645fb77df4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:32.375344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0f7fe553-f253-46d0-85e2-b2645fb77df4 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:32.381982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0f7fe553-f253-46d0-85e2-b2645fb77df4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:32.501599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 878/1759] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:10:32 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:32.640889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b8b686-2028-4720-afa6-666b3508102d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:32.643596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-01b8b686-2028-4720-afa6-666b3508102d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:32.685426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b8b686-2028-4720-afa6-666b3508102d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:32.702258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca68db-7163-42e2-9076-240173838f3c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:32.703353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca68db-7163-42e2-9076-240173838f3c service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:32.709630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca68db-7163-42e2-9076-240173838f3c service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:32.716159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca68db-7163-42e2-9076-240173838f3c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:32.747070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eca68db-7163-42e2-9076-240173838f3c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:32.751994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eca68db-7163-42e2-9076-240173838f3c service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:32.788693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 882/1760] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:32 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:32.890486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e223d04-703e-45a2-82a7-12e0cdce2868 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:32.890715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e223d04-703e-45a2-82a7-12e0cdce2868 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:32.891158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e223d04-703e-45a2-82a7-12e0cdce2868 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:32.893337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 879/1761] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:10:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:33.049400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:33.050214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:33.051663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:33.109549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:33.118024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:33.291215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 883/1762] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:10:33 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:33.341677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e223d04-703e-45a2-82a7-12e0cdce2868 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:33.343952 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2e223d04-703e-45a2-82a7-12e0cdce2868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:33.394681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e223d04-703e-45a2-82a7-12e0cdce2868 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:33.418063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:33.422143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:33.422442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:33.422809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:33.456418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:33.458061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:33.494793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 880/1763] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:33 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:33.701086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:33.702535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:33.725654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2d520b-7e12-4ef3-928e-89c1b19270a4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:33.738734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:33.740949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:33.757770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:33.759065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:33.761124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:33.761566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] PATH_INFO: `/v3/projects/534ca4147149416bb8877e247a2fd6db` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:33.763677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] RBAC: Authorizing `identity:delete_project(project_id=534ca4147149416bb8877e247a2fd6db)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:33.764383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bb5bcf-8cab-46bc-9cd6-a5c99ea7d629 service placement] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:33.767911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:33.777094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:33.778259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:33.778840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:33.779345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] PATH_INFO: `/v3/projects/fe78696ea79041729421265731148164` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:33.782384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe78696ea79041729421265731148164)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:33.785796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:33.892107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe78696ea79041729421265731148164', 'request_id': 'req-5819289b-ee57-4f4a-8892-e8489b63582f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.899726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe78696ea79041729421265731148164', 'request_id': 'req-5819289b-ee57-4f4a-8892-e8489b63582f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.903531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '534ca4147149416bb8877e247a2fd6db', 'request_id': 'req-849b12ab-d6bf-4976-a1d2-f6303661870a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.907383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe78696ea79041729421265731148164', 'request_id': 'req-5819289b-ee57-4f4a-8892-e8489b63582f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.915205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '534ca4147149416bb8877e247a2fd6db', 'request_id': 'req-849b12ab-d6bf-4976-a1d2-f6303661870a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.930268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '534ca4147149416bb8877e247a2fd6db', 'request_id': 'req-849b12ab-d6bf-4976-a1d2-f6303661870a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.931415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] The token cache is being invalidate because project fe78696ea79041729421265731148164 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:33.932367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.935352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:33.936417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5819289b-ee57-4f4a-8892-e8489b63582f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.940231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 881/1764] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:33 2022] DELETE /identity/v3/projects/fe78696ea79041729421265731148164 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:33.945767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c901e-fa42-4495-a9f4-c3d47fe32c74 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:33.946932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c901e-fa42-4495-a9f4-c3d47fe32c74 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:33.948855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75c901e-fa42-4495-a9f4-c3d47fe32c74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:33.952283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] The token cache is being invalidate because project 534ca4147149416bb8877e247a2fd6db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:33.953860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.954757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-849b12ab-d6bf-4976-a1d2-f6303661870a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:33.956875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 884/1765] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:33 2022] DELETE /identity/v3/projects/534ca4147149416bb8877e247a2fd6db => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:33.962436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:33.962921 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:33.963769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:33.978464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a75c901e-fa42-4495-a9f4-c3d47fe32c74 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:33.980301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a75c901e-fa42-4495-a9f4-c3d47fe32c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:33.988997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:33.990649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.073302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:34.073668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 885/1766] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:34.074721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:34.081969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.100315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95678d34-0be7-4d23-9cbd-cf517e3ffae4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:34.122849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:34.124255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.125643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.125804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 882/1767] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:34.127768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96da74cd-a5b2-4c66-9fd6-0277f36e026a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.128322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] PATH_INFO: `/v3/users/f55fd1108d15468489d5c4ba958fb1fd` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.130096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] RBAC: Authorizing `identity:delete_user(user_id=f55fd1108d15468489d5c4ba958fb1fd)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:34.131427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96da74cd-a5b2-4c66-9fd6-0277f36e026a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.132497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96da74cd-a5b2-4c66-9fd6-0277f36e026a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.134359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:34.166080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-96da74cd-a5b2-4c66-9fd6-0277f36e026a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:34.167655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-96da74cd-a5b2-4c66-9fd6-0277f36e026a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.183699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 883/1768] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:34.185569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02015a05-dc31-492f-ad7f-fc42366f90db None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.185998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02015a05-dc31-492f-ad7f-fc42366f90db None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.186471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02015a05-dc31-492f-ad7f-fc42366f90db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.221474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-02015a05-dc31-492f-ad7f-fc42366f90db None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:34.228459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-02015a05-dc31-492f-ad7f-fc42366f90db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.281761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 884/1769] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:34.281945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f55fd1108d15468489d5c4ba958fb1fd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:34.281945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f55fd1108d15468489d5c4ba958fb1fd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:34.282095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02015a05-dc31-492f-ad7f-fc42366f90db None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:34.299162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f55fd1108d15468489d5c4ba958fb1fd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:34.302981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f55fd1108d15468489d5c4ba958fb1fd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:34.316167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02015a05-dc31-492f-ad7f-fc42366f90db None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:34.316963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f55fd1108d15468489d5c4ba958fb1fd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:34.331950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d60105-bd90-4e2e-b377-8c7e2cbef627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:34.333002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d60105-bd90-4e2e-b377-8c7e2cbef627 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.334881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d60105-bd90-4e2e-b377-8c7e2cbef627 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.335165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d60105-bd90-4e2e-b377-8c7e2cbef627 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.339292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d60105-bd90-4e2e-b377-8c7e2cbef627 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:34.339631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d60105-bd90-4e2e-b377-8c7e2cbef627 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:34.342183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d60105-bd90-4e2e-b377-8c7e2cbef627 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:34.358156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 886/1770] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:34 2022] DELETE /identity/v3/users/f55fd1108d15468489d5c4ba958fb1fd => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:34.358156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:34.358156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.372924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 885/1771] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:34.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.375281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.398596 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:34.412738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.516221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 886/1772] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:34.521177 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:34.525947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.542427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc77086-5b50-48bd-a972-048bcbd6772f admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:34.555377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:34.556785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.557302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.557706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.558643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:34.558958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:34.568133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:34.569320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81802913-1638-47fe-832c-54d20c89f4b7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:34.574996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:34.575681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.576467 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.576860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.579851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:34.587387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:34.625055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 887/1773] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:34.631856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:34.637565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 887/1774] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:34 2022] POST /identity/v3/projects => generated 381 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:34.638676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:34.645705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:34.654931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:34.777702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380199cc-77f2-4947-9181-e04f4196f287 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:34.870181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:34.870870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.871348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.871484 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.892415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:34.892415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:34.901618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 888/1775] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:34.903081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:34.907966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037c4891-f95d-4e63-9a3f-2b2612e25206 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:34.923870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:34.923870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.923870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.924431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.926840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:34.931370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8969f53c-cb7b-4fec-b528-79d8c5de7973 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:34.932295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:34.944808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:34.947647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:34.948651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:34.948942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:34.954505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:34.959557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.027385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 888/1776] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:34 2022] POST /identity/v3/projects => generated 365 bytes in 389 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:35.033156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 889/1777] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:34 2022] POST /identity/v3/users => generated 291 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:35.034203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.039152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.064606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2774bdb2-5194-4f23-b670-8ef40e67e7eb admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.156646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:35.176925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.180617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.182476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.182790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.183562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.188150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.228350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 890/1778] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:35.231675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.263038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10421d0f-3744-47f5-917c-8d90c8822aff admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.279554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.280043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.280334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.280702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.283675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.285348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2578c6-9bbe-49b9-ad15-6cafaffbbb2a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.289594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.299762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.300766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.307170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.307568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.308649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.321774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.339885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 891/1779] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:35.344550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.383838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39112e8d-b37e-40de-9430-d59f31613902 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.397907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f76c45b-30b6-4317-9028-f1deace11c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.399776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 889/1780] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:35 2022] POST /identity/v3/users => generated 283 bytes in 366 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:35.400526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f76c45b-30b6-4317-9028-f1deace11c85 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.400633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.401133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f76c45b-30b6-4317-9028-f1deace11c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.401587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f76c45b-30b6-4317-9028-f1deace11c85 admin admin] PATH_INFO: `/v3/projects/a7deda2682de45eebe55f0050d5b2014/users/4f5225202d2d408abb981f1fe28fa4fe/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.427232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4a2a4-07d2-4c42-bd0f-735e6fe2f249 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.440083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f76c45b-30b6-4317-9028-f1deace11c85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7deda2682de45eebe55f0050d5b2014, user_id=4f5225202d2d408abb981f1fe28fa4fe, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.446145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f76c45b-30b6-4317-9028-f1deace11c85 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.454433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.459383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.460072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.460194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.470219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.475768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.483070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 892/1781] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:35 2022] PUT /identity/v3/projects/a7deda2682de45eebe55f0050d5b2014/users/4f5225202d2d408abb981f1fe28fa4fe/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:35.498067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6420cfdf-b925-4d8d-a7c9-241ea1b7feb1 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.498397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6420cfdf-b925-4d8d-a7c9-241ea1b7feb1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.498624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6420cfdf-b925-4d8d-a7c9-241ea1b7feb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.537615 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 890/1782] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:35.538894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.546843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:35.575144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a7d253-210a-4c59-bbe5-74476f130273 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.597763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6420cfdf-b925-4d8d-a7c9-241ea1b7feb1 None None] MFA Rules not processed for user `4f5225202d2d408abb981f1fe28fa4fe`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:35.600048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6420cfdf-b925-4d8d-a7c9-241ea1b7feb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:35.782753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.783454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.791761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.792087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.792910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.796532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.817520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 893/1783] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:35.822033 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6420cfdf-b925-4d8d-a7c9-241ea1b7feb1 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.828933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 891/1784] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:35 2022] GET /identity/v3/roles => generated 1420 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:35.829830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.843153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6420cfdf-b925-4d8d-a7c9-241ea1b7feb1 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.851687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fdc7da-8001-48c9-a8be-791eeea4a94b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:35.855505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.856770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.857246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.857540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.858889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.860879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.865897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c384337-ecc7-4e57-b398-9730e16d24fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:35.866816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c384337-ecc7-4e57-b398-9730e16d24fa admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.867102 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c384337-ecc7-4e57-b398-9730e16d24fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.868114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c384337-ecc7-4e57-b398-9730e16d24fa admin admin] PATH_INFO: `/v3/projects/29148069958f465dbb9cc395d8eb8b42/users/4cf8a7057e784e74ab54971741f1ab6d/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:35.904378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c384337-ecc7-4e57-b398-9730e16d24fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=29148069958f465dbb9cc395d8eb8b42, user_id=4cf8a7057e784e74ab54971741f1ab6d, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:35.908256 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c384337-ecc7-4e57-b398-9730e16d24fa admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:35.937846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 894/1785] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:35 2022] POST /identity/v3/projects => generated 381 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:35.941972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:35.969890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 892/1786] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:35 2022] PUT /identity/v3/projects/29148069958f465dbb9cc395d8eb8b42/users/4cf8a7057e784e74ab54971741f1ab6d/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:35.971639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6097c01-fdc7-47dd-b72d-5e9c7ec15be6 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:35.971953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6097c01-fdc7-47dd-b72d-5e9c7ec15be6 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:35.972225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6097c01-fdc7-47dd-b72d-5e9c7ec15be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.038299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b6097c01-fdc7-47dd-b72d-5e9c7ec15be6 None None] MFA Rules not processed for user `4cf8a7057e784e74ab54971741f1ab6d`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:36.040212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b6097c01-fdc7-47dd-b72d-5e9c7ec15be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:36.146389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc408ac6-880e-4bff-868e-c5dc7afb6724 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.188079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 893/1787] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:35 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:36.193547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6097c01-fdc7-47dd-b72d-5e9c7ec15be6 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.207211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:36.250116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.251392 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.251819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.252444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.253281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.257158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.307491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 895/1788] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:35 2022] POST /identity/v3/users => generated 297 bytes in 365 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:36.308858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.329514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c652d6-86f6-4c07-9c8a-d4881e9c0d89 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.337760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.339102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.339498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.340010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.341711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.344751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.348540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:36.371947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 896/1789] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:36.373150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.374124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6097c01-fdc7-47dd-b72d-5e9c7ec15be6 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.391026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83eaab2-c9c6-42f0-9ed3-18847861a556 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.397399 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.399054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.399823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.400285 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.401164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.401600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.402398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.403047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] PATH_INFO: `/v3/projects/bcae2d3aae7b47e08f985c56c18dda7a/users/4bf26700c704482eaeaa18587074f50d/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.403794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.407054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.431113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcae2d3aae7b47e08f985c56c18dda7a, user_id=4bf26700c704482eaeaa18587074f50d, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.436176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.464947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 897/1790] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:36 2022] PUT /identity/v3/projects/bcae2d3aae7b47e08f985c56c18dda7a/users/4bf26700c704482eaeaa18587074f50d/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:36.465344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 894/1791] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:36 2022] POST /identity/v3/projects => generated 365 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:36.469513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.469649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.628166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102ecef8-4449-4e29-9592-d21c4aea4cae admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.644016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.645474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.647189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.648018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.650569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.650870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68bd0b4-822f-4534-a75b-01c622b2add8 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.658285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.690605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.691308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.691586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.691873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.693307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.695066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.718618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 898/1792] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:36 2022] POST /identity/v3/users => generated 289 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:36.722870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.752075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 895/1793] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:36.754005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02837c3f-d33b-47a5-b188-c24f6073bd80 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.756767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.772784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.773605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.774246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.774628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.775489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.779028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14adfc36-c4e7-479e-87d4-e1684f9f779c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.779361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.804779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b92957-45c0-49e6-b417-972693445231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.805705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b92957-45c0-49e6-b417-972693445231 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.805997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b92957-45c0-49e6-b417-972693445231 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.806281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b92957-45c0-49e6-b417-972693445231 admin admin] PATH_INFO: `/v3/projects/bcae2d3aae7b47e08f985c56c18dda7a/users/4bf26700c704482eaeaa18587074f50d/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.811527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b92957-45c0-49e6-b417-972693445231 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcae2d3aae7b47e08f985c56c18dda7a, user_id=4bf26700c704482eaeaa18587074f50d, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.812128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 899/1794] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:36.816636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:36.820890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b92957-45c0-49e6-b417-972693445231 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.833319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033d72c7-6442-4041-8a3b-6e1675bb9740 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:36.851422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 896/1795] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:36 2022] PUT /identity/v3/projects/bcae2d3aae7b47e08f985c56c18dda7a/users/4bf26700c704482eaeaa18587074f50d/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:36.859870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:36.862974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.863850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:36.864178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.864502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.865540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:36.865918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] PATH_INFO: `/v3/projects/c5b6419cb92a489eaa5df4159c6d2c68/users/4b96799b4844482b88ac06f6f76116d3/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:36.916763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5b6419cb92a489eaa5df4159c6d2c68, user_id=4b96799b4844482b88ac06f6f76116d3, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:36.923944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:36.946619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] MFA Rules not processed for user `4bf26700c704482eaeaa18587074f50d`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:36.948427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:36.955710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 900/1796] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:36 2022] PUT /identity/v3/projects/c5b6419cb92a489eaa5df4159c6d2c68/users/4b96799b4844482b88ac06f6f76116d3/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:36.960960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:37.061968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07715fed-9298-410f-9613-227b5d21478a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:37.069278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:37.070390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.071059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.071836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.072919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:37.075838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:37.086346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:37.095051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 901/1797] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:36 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:37.101691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:37.116621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 897/1798] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:36 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:10:37.116908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82454a47-bb36-4964-84ce-0908af3b6198 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:37.124022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:37.125983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:37.133862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:37.135750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.136195 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.136760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] PATH_INFO: `/v3/projects/c5b6419cb92a489eaa5df4159c6d2c68/users/4b96799b4844482b88ac06f6f76116d3/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.139941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5b6419cb92a489eaa5df4159c6d2c68, user_id=4b96799b4844482b88ac06f6f76116d3, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:37.145980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:37.148906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0637537-c9ad-44a7-85e5-145dba6b0624 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:37.160924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 902/1799] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:37 2022] PUT /identity/v3/projects/c5b6419cb92a489eaa5df4159c6d2c68/users/4b96799b4844482b88ac06f6f76116d3/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:37.162289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:37.165524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:37.165769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:37.167410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.167967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.168538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] PATH_INFO: `/v3/users/399d3739d12947428e45ca64b3b71dea` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.170769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] RBAC: Authorizing `identity:delete_user(user_id=399d3739d12947428e45ca64b3b71dea)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:37.173011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:37.200322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:37.270867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbae595-9b68-4509-ad18-2547a6f0141d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:37.272713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '399d3739d12947428e45ca64b3b71dea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:37.276151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:37.277100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.277898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.278316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.282909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '399d3739d12947428e45ca64b3b71dea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:37.284092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '399d3739d12947428e45ca64b3b71dea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:37.285330 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '399d3739d12947428e45ca64b3b71dea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:37.305113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4c83dec7-424e-4cec-8520-32d71caa4ddc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '399d3739d12947428e45ca64b3b71dea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:37.327754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 898/1800] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:37 2022] DELETE /identity/v3/users/399d3739d12947428e45ca64b3b71dea => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:37.331216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.331525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.331819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.375757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] MFA Rules not processed for user `4b96799b4844482b88ac06f6f76116d3`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:37.377384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:37.479714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:37.481215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:37.502253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 903/1801] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:37 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:37.519205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:37.524732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:37.547247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 899/1802] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:37 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:37.565334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:37.567508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:37.692571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbe976a-41ca-4993-b9cb-5deb7920a0fa None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:37.697291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1a034e-6330-4c89-a0f5-2a74ee49b8f0 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:37.698206 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc3d548-f555-46e3-9fe2-889c3cdbb197 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:37.698999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc3d548-f555-46e3-9fe2-889c3cdbb197 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.699502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc3d548-f555-46e3-9fe2-889c3cdbb197 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.699882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc3d548-f555-46e3-9fe2-889c3cdbb197 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.711562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:37.715074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc3d548-f555-46e3-9fe2-889c3cdbb197 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:37.717712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc3d548-f555-46e3-9fe2-889c3cdbb197 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:37.759935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 900/1803] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:37 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:37.762280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a4752-37a0-4f13-a0d2-577f19615dec None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.762876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a4752-37a0-4f13-a0d2-577f19615dec None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.763450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a4752-37a0-4f13-a0d2-577f19615dec None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.766020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 901/1804] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:10:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:37.766973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a4752-37a0-4f13-a0d2-577f19615dec None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:37.769385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-544a4752-37a0-4f13-a0d2-577f19615dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:37.799936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a4752-37a0-4f13-a0d2-577f19615dec None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:37.806616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be63fe7-0c50-459a-92d8-2fb0cbbe9c8d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:37.807398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be63fe7-0c50-459a-92d8-2fb0cbbe9c8d service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.807899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be63fe7-0c50-459a-92d8-2fb0cbbe9c8d service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.808217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be63fe7-0c50-459a-92d8-2fb0cbbe9c8d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.811700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9be63fe7-0c50-459a-92d8-2fb0cbbe9c8d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:37.834834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:37.835767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:37.836029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:37.836308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] PATH_INFO: `/v3/projects/ceb99c8505a549a2976bde811ad29d14` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:37.837848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ceb99c8505a549a2976bde811ad29d14)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:37.840973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:37.857640 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:37.971227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be63fe7-0c50-459a-92d8-2fb0cbbe9c8d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:37.975118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be63fe7-0c50-459a-92d8-2fb0cbbe9c8d service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:37.981597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ceb99c8505a549a2976bde811ad29d14', 'request_id': 'req-e6a00e69-b94d-49a7-8ada-17078ea85b87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:37.995528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 902/1805] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:37 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:37.996820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ceb99c8505a549a2976bde811ad29d14', 'request_id': 'req-e6a00e69-b94d-49a7-8ada-17078ea85b87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:37.998256 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d165d498-cd55-4040-8369-a0133781535d None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:38.000068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d165d498-cd55-4040-8369-a0133781535d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:38.000730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d165d498-cd55-4040-8369-a0133781535d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:38.019929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ceb99c8505a549a2976bde811ad29d14', 'request_id': 'req-e6a00e69-b94d-49a7-8ada-17078ea85b87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.040331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] The token cache is being invalidate because project ceb99c8505a549a2976bde811ad29d14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:38.042038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.043788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.050893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d165d498-cd55-4040-8369-a0133781535d None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:38.052384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 904/1806] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:37 2022] DELETE /identity/v3/projects/ceb99c8505a549a2976bde811ad29d14 => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:38.054060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:38.055397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d165d498-cd55-4040-8369-a0133781535d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:38.060172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:38.221338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a00e69-b94d-49a7-8ada-17078ea85b87 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:38.230804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 903/1807] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:10:37 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:38.232156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d165d498-cd55-4040-8369-a0133781535d None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:38.235280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d165d498-cd55-4040-8369-a0133781535d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:38.249397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:38.252383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:38.252856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:38.253916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] PATH_INFO: `/v3/users/d7b549b96cab49eda16ebb026666f8d2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:38.256816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7b549b96cab49eda16ebb026666f8d2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:38.259573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:38.272021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d165d498-cd55-4040-8369-a0133781535d None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:38.284783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:38.285780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:38.286936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:38.287877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] PATH_INFO: `/v3/projects/876a37cc5c744c38a6c1da63aca578ab` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:38.290135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=876a37cc5c744c38a6c1da63aca578ab)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:38.294395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:38.388455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7b549b96cab49eda16ebb026666f8d2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.390082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7b549b96cab49eda16ebb026666f8d2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.404286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7b549b96cab49eda16ebb026666f8d2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.405130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7b549b96cab49eda16ebb026666f8d2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.417949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '876a37cc5c744c38a6c1da63aca578ab', 'request_id': 'req-266a827b-425c-4e62-afde-e3ac0b3151d0'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.435307 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7b549b96cab49eda16ebb026666f8d2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.435479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '876a37cc5c744c38a6c1da63aca578ab', 'request_id': 'req-266a827b-425c-4e62-afde-e3ac0b3151d0'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.452370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '876a37cc5c744c38a6c1da63aca578ab', 'request_id': 'req-266a827b-425c-4e62-afde-e3ac0b3151d0'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.476895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 905/1808] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:38 2022] DELETE /identity/v3/users/d7b549b96cab49eda16ebb026666f8d2 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:38.482453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] The token cache is being invalidate because project 876a37cc5c744c38a6c1da63aca578ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:38.482826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.484331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:38.484788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.486911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 904/1809] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:38 2022] DELETE /identity/v3/projects/876a37cc5c744c38a6c1da63aca578ab => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:38.487871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:38.501725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:38.503159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:38.522751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:38.571845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:38.686412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb86e6c2-944c-42f5-82f0-cb45175db211 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:38.693893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:38.694815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:38.695285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:38.695721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:38.699028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:38.706784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266a827b-425c-4e62-afde-e3ac0b3151d0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:38.744992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:38.748840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:38.774571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 906/1810] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:38 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:38.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:38.778003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:38.837662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:38.838965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:38.839534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:38.840191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] PATH_INFO: `/v3/users/4b04d8ffacb34c89b2c8110009867cf6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:38.842432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b04d8ffacb34c89b2c8110009867cf6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:38.844764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:38.908346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbb25da-3acd-4184-9b41-fd3246ac80ad service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:38.916952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:38.918398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:38.920696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:38.925137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] PATH_INFO: `/v3/users/4cf8a7057e784e74ab54971741f1ab6d` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:38.928303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cf8a7057e784e74ab54971741f1ab6d)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:38.931665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:38.942451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b04d8ffacb34c89b2c8110009867cf6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.952593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b04d8ffacb34c89b2c8110009867cf6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.953810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b04d8ffacb34c89b2c8110009867cf6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.955162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b04d8ffacb34c89b2c8110009867cf6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.970507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-471692bc-4e13-4d76-9520-75c46d45a033 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b04d8ffacb34c89b2c8110009867cf6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.986054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 905/1811] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:38 2022] DELETE /identity/v3/users/4b04d8ffacb34c89b2c8110009867cf6 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:38.995387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cf8a7057e784e74ab54971741f1ab6d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:38.996080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cf8a7057e784e74ab54971741f1ab6d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.003936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cf8a7057e784e74ab54971741f1ab6d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.004648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cf8a7057e784e74ab54971741f1ab6d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.013737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cf8a7057e784e74ab54971741f1ab6d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.038010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 907/1812] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:38 2022] DELETE /identity/v3/users/4cf8a7057e784e74ab54971741f1ab6d => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:39.042508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.043017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.043400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.062358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:39.063590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:39.100384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:39.150942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 906/1813] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:39.177513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:39.179141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:39.211336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:39.263507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5221181c-bd47-4476-b3fb-98078bd36dc8 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:39.270390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:39.271916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.272365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.272871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.287716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:39.289738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:39.303186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 908/1814] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:39.327355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:39.328966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:39.342748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0839c961-2e67-4f32-ad72-b37d1f8a422f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:39.353583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:39.355777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.356351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.357025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] PATH_INFO: `/v3/projects/f128e133c7264725bb1a89f59964e0cf` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.359036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f128e133c7264725bb1a89f59964e0cf)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:39.362204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:39.497902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f128e133c7264725bb1a89f59964e0cf', 'request_id': 'req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.509723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f128e133c7264725bb1a89f59964e0cf', 'request_id': 'req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.509973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:39.511472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:39.520993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f128e133c7264725bb1a89f59964e0cf', 'request_id': 'req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.550383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1231764b-f712-424c-b9d5-62e82d85c9f5 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:39.565260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] The token cache is being invalidate because project f128e133c7264725bb1a89f59964e0cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:39.565562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.567644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1bb0d5a4-3794-4d7b-bc73-0c64dc7f88dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.570901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 907/1815] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:39 2022] DELETE /identity/v3/projects/f128e133c7264725bb1a89f59964e0cf => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:39.575718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fecf974-dbab-41a5-97fb-a09863d09937 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.576009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fecf974-dbab-41a5-97fb-a09863d09937 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.576287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fecf974-dbab-41a5-97fb-a09863d09937 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.611412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0fecf974-dbab-41a5-97fb-a09863d09937 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:39.617033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0fecf974-dbab-41a5-97fb-a09863d09937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:39.643708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:39.645720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.646484 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.649584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] PATH_INFO: `/v3/projects/4103d681bda344e2ab20233b677d40fb` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.652249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4103d681bda344e2ab20233b677d40fb)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:39.655053 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:39.716376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 908/1816] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:39.718635 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246de69-43a0-4319-ac7b-af9c7575233f None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.718965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246de69-43a0-4319-ac7b-af9c7575233f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.719305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246de69-43a0-4319-ac7b-af9c7575233f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.756337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4103d681bda344e2ab20233b677d40fb', 'request_id': 'req-921bdd21-c02e-4ed9-9416-545e36499b19'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.764549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4103d681bda344e2ab20233b677d40fb', 'request_id': 'req-921bdd21-c02e-4ed9-9416-545e36499b19'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.767954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-2246de69-43a0-4319-ac7b-af9c7575233f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:39.770897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2246de69-43a0-4319-ac7b-af9c7575233f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:39.776172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4103d681bda344e2ab20233b677d40fb', 'request_id': 'req-921bdd21-c02e-4ed9-9416-545e36499b19'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.795676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] The token cache is being invalidate because project 4103d681bda344e2ab20233b677d40fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:39.796097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.796893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-921bdd21-c02e-4ed9-9416-545e36499b19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:39.798766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 909/1817] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:39 2022] DELETE /identity/v3/projects/4103d681bda344e2ab20233b677d40fb => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:39.804989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc7b3f-e7c8-4041-a0dd-b823f89e0747 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.805334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc7b3f-e7c8-4041-a0dd-b823f89e0747 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.805621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbc7b3f-e7c8-4041-a0dd-b823f89e0747 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.829843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7dbc7b3f-e7c8-4041-a0dd-b823f89e0747 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:39.831229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7dbc7b3f-e7c8-4041-a0dd-b823f89e0747 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:39.897483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:39.914247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:39.938202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 910/1818] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:39.939634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc7b3f-e7c8-4041-a0dd-b823f89e0747 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:39.941905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 909/1819] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:39.946254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433537d-3fd6-4038-9311-b247f928fa46 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.948352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433537d-3fd6-4038-9311-b247f928fa46 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.948672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433537d-3fd6-4038-9311-b247f928fa46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.965040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbc7b3f-e7c8-4041-a0dd-b823f89e0747 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:39.977998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:39.978761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:39.980615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:39.980974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:39.981814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:39.982913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:39.987105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:39.994039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-3433537d-3fd6-4038-9311-b247f928fa46 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:39.996493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3433537d-3fd6-4038-9311-b247f928fa46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.014505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 910/1820] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:40.015959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3433537d-3fd6-4038-9311-b247f928fa46 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.018047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 911/1821] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:40.023076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.048440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3433537d-3fd6-4038-9311-b247f928fa46 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.055720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e6ece3-bd61-477b-aaf1-a0a00494e311 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.057257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.058459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.058815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.059113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.060209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:40.060613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.064895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.073250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.073978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.074385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.074822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.076291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.084043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.094160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 911/1822] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:40.104731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.122529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 912/1823] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:40 2022] POST /identity/v3/projects => generated 375 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:40.123846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.125762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.149569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4fb71f-e917-49a8-a2d5-69eca0c46abc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.278965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.279628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.279917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.280197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.280949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.282869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.296542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cdd989-343d-46af-8bda-1f377e6c2939 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.308351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.310727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.311303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.312364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] PATH_INFO: `/v3/projects/29148069958f465dbb9cc395d8eb8b42` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.314815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] RBAC: Authorizing `identity:delete_project(project_id=29148069958f465dbb9cc395d8eb8b42)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.317302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.322993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 912/1824] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:40 2022] POST /identity/v3/projects => generated 383 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:40.322993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.383281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29148069958f465dbb9cc395d8eb8b42', 'request_id': 'req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.389546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29148069958f465dbb9cc395d8eb8b42', 'request_id': 'req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.395464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b783b-33f6-46cd-9695-4a209e00c24a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.399693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29148069958f465dbb9cc395d8eb8b42', 'request_id': 'req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.424459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] The token cache is being invalidate because project 29148069958f465dbb9cc395d8eb8b42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:40.424765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.426106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.428884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 913/1825] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:40 2022] DELETE /identity/v3/projects/29148069958f465dbb9cc395d8eb8b42 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:40.430404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.432815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.483673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.484621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.484912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.485196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.485901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.487994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.498624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:40.517162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb957c7-2ee7-4f60-8ba4-8e2508d14cad admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.524446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 913/1826] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:40 2022] POST /identity/v3/users => generated 288 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:40.524446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.526590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.531441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.533079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.533438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.533975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.534113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:40.534817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.537905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.543944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f21d96-5645-42d2-996e-bac2b24549d4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.549991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.550986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.551373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.551748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] PATH_INFO: `/v3/users/4b96799b4844482b88ac06f6f76116d3` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.553121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b96799b4844482b88ac06f6f76116d3)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.555109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.569095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 914/1827] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:40 2022] POST /identity/v3/users => generated 292 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:40.570023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.572239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.600816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cba529-d01f-434d-8432-cf860df0892b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.605923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.606881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.607368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.608012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.609005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.610870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.619864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b96799b4844482b88ac06f6f76116d3'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.622614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 915/1828] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:40.623662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.631882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b96799b4844482b88ac06f6f76116d3'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.632458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b96799b4844482b88ac06f6f76116d3'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.633158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b96799b4844482b88ac06f6f76116d3'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.635557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686274c4-f643-4780-b696-528a9e9b46cb admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.642085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.643223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.643900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.644359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.645246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.655786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.656066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b96799b4844482b88ac06f6f76116d3'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:40.661584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 916/1829] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:40.662569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.664614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.673174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 914/1830] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:40 2022] DELETE /identity/v3/users/4b96799b4844482b88ac06f6f76116d3 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:40.673846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.675164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.683979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db67cc-a0bb-4329-a550-7f0c9925708c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.759464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.760149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.760457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.760739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.761483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.763130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.774207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 917/1831] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:40.776646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c1a0a-6d46-478c-9be0-15ea75cc5ad7 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.778566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.784212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.785003 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.785339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.785689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.786622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.788584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7c2da4-28f7-4395-a39d-af32316dcaee admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.789371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.793493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df2b469-00fc-4e46-be03-b2f61fb09c77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.794384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2b469-00fc-4e46-be03-b2f61fb09c77 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.794672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2b469-00fc-4e46-be03-b2f61fb09c77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.794949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2b469-00fc-4e46-be03-b2f61fb09c77 admin admin] PATH_INFO: `/v3/projects/e21303cf10464bdf9e2936db97f86792/users/980825b12301415197171fcabcbac199/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.805034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 915/1832] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:40 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:40.806132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.812429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df2b469-00fc-4e46-be03-b2f61fb09c77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e21303cf10464bdf9e2936db97f86792, user_id=980825b12301415197171fcabcbac199, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.814673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df2b469-00fc-4e46-be03-b2f61fb09c77 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.818811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e760e80b-f412-4be3-a4f4-ff3b87efd22c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:40.830371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 918/1833] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:40 2022] PUT /identity/v3/projects/e21303cf10464bdf9e2936db97f86792/users/980825b12301415197171fcabcbac199/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:40.830810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a9f5d7-8cad-4783-be47-eb7b05859ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:40.832168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9f5d7-8cad-4783-be47-eb7b05859ba1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.832673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9f5d7-8cad-4783-be47-eb7b05859ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.836690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a9f5d7-8cad-4783-be47-eb7b05859ba1 admin admin] PATH_INFO: `/v3/projects/001d710a50b943808aa74cb9a17b25a7/users/1b47385acd1644aa98f1a30b24a75b59/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.841030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190610d4-3e37-4323-aef1-10e698b8636c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.841359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190610d4-3e37-4323-aef1-10e698b8636c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.841563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190610d4-3e37-4323-aef1-10e698b8636c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.854226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a9f5d7-8cad-4783-be47-eb7b05859ba1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=001d710a50b943808aa74cb9a17b25a7, user_id=1b47385acd1644aa98f1a30b24a75b59, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:40.857090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a9f5d7-8cad-4783-be47-eb7b05859ba1 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:40.872607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-190610d4-3e37-4323-aef1-10e698b8636c None None] MFA Rules not processed for user `980825b12301415197171fcabcbac199`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:40.874653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-190610d4-3e37-4323-aef1-10e698b8636c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.885311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 916/1834] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:40 2022] PUT /identity/v3/projects/001d710a50b943808aa74cb9a17b25a7/users/1b47385acd1644aa98f1a30b24a75b59/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:40.887462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406cd8c9-85c9-4898-a549-7d4920f77c6e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:40.887801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406cd8c9-85c9-4898-a549-7d4920f77c6e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:40.888117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406cd8c9-85c9-4898-a549-7d4920f77c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:40.926881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-406cd8c9-85c9-4898-a549-7d4920f77c6e None None] MFA Rules not processed for user `1b47385acd1644aa98f1a30b24a75b59`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:40.929730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-406cd8c9-85c9-4898-a549-7d4920f77c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:40.974450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 919/1835] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:40 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:40.978789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190610d4-3e37-4323-aef1-10e698b8636c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:40.991208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:41.008085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:41.030101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 917/1836] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:10:41.034597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406cd8c9-85c9-4898-a549-7d4920f77c6e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.059366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190610d4-3e37-4323-aef1-10e698b8636c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.064944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.065879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.066173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.066456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.067153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.070294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.108733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 920/1837] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:40 2022] POST /identity/v3/projects => generated 375 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:41.109912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.113058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:41.119420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406cd8c9-85c9-4898-a549-7d4920f77c6e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.184205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2146000f-d13f-4ffe-9e27-ee1be4652bb9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.184804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.185476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.185760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.186043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.186725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.188765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.188973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.189832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.190121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.190402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] PATH_INFO: `/v3/projects/c5b6419cb92a489eaa5df4159c6d2c68` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.191579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5b6419cb92a489eaa5df4159c6d2c68)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.193550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.218707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 918/1838] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:41 2022] POST /identity/v3/projects => generated 383 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:41.218707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.263998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5b6419cb92a489eaa5df4159c6d2c68', 'request_id': 'req-e9e066f5-0d00-49e4-8f69-44342b2cf669'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:41.276606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5b6419cb92a489eaa5df4159c6d2c68', 'request_id': 'req-e9e066f5-0d00-49e4-8f69-44342b2cf669'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:41.295260 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5b6419cb92a489eaa5df4159c6d2c68', 'request_id': 'req-e9e066f5-0d00-49e4-8f69-44342b2cf669'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:41.308327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] The token cache is being invalidate because project c5b6419cb92a489eaa5df4159c6d2c68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:41.308665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:41.309326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:41.313984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 921/1839] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:41 2022] DELETE /identity/v3/projects/c5b6419cb92a489eaa5df4159c6d2c68 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:41.313984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.313984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:41.339204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38dbc6a-ab9b-4856-9369-729836dea498 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.370378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:41.371885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:41.414190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e066f5-0d00-49e4-8f69-44342b2cf669 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.423161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81df1cc4-4ba2-447b-ba52-a85b4d780ab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.424749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81df1cc4-4ba2-447b-ba52-a85b4d780ab5 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.425429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81df1cc4-4ba2-447b-ba52-a85b4d780ab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.425971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81df1cc4-4ba2-447b-ba52-a85b4d780ab5 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.427044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81df1cc4-4ba2-447b-ba52-a85b4d780ab5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.432096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81df1cc4-4ba2-447b-ba52-a85b4d780ab5 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.475401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 922/1840] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:41 2022] POST /identity/v3/users => generated 298 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:41.484355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.485568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.486107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.489454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.490987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.491335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.491718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.491994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.494612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.527984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:41.529298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:41.533388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 919/1841] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:41 2022] POST /identity/v3/users => generated 294 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:41.535019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.549659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 923/1842] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:41.549659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.549659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:41.554108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234bc793-e1fa-4c01-a2fd-4d7a1a5c237d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.561886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2b5f49-2209-4772-820a-0a9dd9811572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.563124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2b5f49-2209-4772-820a-0a9dd9811572 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.563472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2b5f49-2209-4772-820a-0a9dd9811572 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.563810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2b5f49-2209-4772-820a-0a9dd9811572 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.564805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2b5f49-2209-4772-820a-0a9dd9811572 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.567030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235c5639-1535-4e50-8e07-f2b4383a8a9a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.567197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2b5f49-2209-4772-820a-0a9dd9811572 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.572786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.573687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.574139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.574303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.574963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.577113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.583245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 920/1843] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:41.585684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2069c-98d5-4e7d-9756-8f7ca2930d0e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.586153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2069c-98d5-4e7d-9756-8f7ca2930d0e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.586658 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf2069c-98d5-4e7d-9756-8f7ca2930d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.592095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 924/1844] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:41.593313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.608299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dd7ea7-7ab3-4551-840b-65ba9a406c83 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.611506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-dcf2069c-98d5-4e7d-9756-8f7ca2930d0e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:41.612843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dcf2069c-98d5-4e7d-9756-8f7ca2930d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:41.613038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.613978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.614263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.614543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] PATH_INFO: `/v3/projects/92f70eacbcca49db85a9d3824786aad6/users/0c4db850abc243c887ee39b39baea210/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.624301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 921/1845] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:41.624301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf2069c-98d5-4e7d-9756-8f7ca2930d0e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.631608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] RBAC: Authorizing `identity:create_grant(project_id=92f70eacbcca49db85a9d3824786aad6, user_id=0c4db850abc243c887ee39b39baea210, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.631986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf2069c-98d5-4e7d-9756-8f7ca2930d0e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.634550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.636875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.637892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.638007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.638188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] PATH_INFO: `/v3/projects/1b907107c8f8461d92b6a2daf658c9ee/users/d7a6bf0045f94f64b4ef8ad9e0164e93/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.651557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 925/1846] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:41 2022] PUT /identity/v3/projects/92f70eacbcca49db85a9d3824786aad6/users/0c4db850abc243c887ee39b39baea210/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:41.653466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.677853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b907107c8f8461d92b6a2daf658c9ee, user_id=d7a6bf0045f94f64b4ef8ad9e0164e93, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.682304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.712475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 922/1847] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:41 2022] PUT /identity/v3/projects/1b907107c8f8461d92b6a2daf658c9ee/users/d7a6bf0045f94f64b4ef8ad9e0164e93/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:41.716236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.745477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7121175f-a645-4ba0-aff9-3cc03c383deb admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.817113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.817877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.818253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.819028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.819397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:41.819896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.821635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.823728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa51ebf-56d8-4c52-807f-85297c01dd90 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.829323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.830342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.831129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.831609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.832660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.834982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.839810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 926/1848] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:41.839810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.848739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d3a062-826f-4d8d-b5a0-3e46fb9b1ede admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.851875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 923/1849] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:41.853782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.858476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.859844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.860142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.860474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.861341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.863085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.868179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3911e2-4f85-4386-bc80-2240d2b4489f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.874097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.875270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.875923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.876505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.877233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 927/1850] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:41 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:41.877653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.878185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.886329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.890088 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d7aa13-fef1-4c65-b349-63166b9aeddd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:41.897556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:41.898558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:41.898845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:41.899124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] PATH_INFO: `/v3/projects/92f70eacbcca49db85a9d3824786aad6/users/0c4db850abc243c887ee39b39baea210/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:41.901380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] RBAC: Authorizing `identity:create_grant(project_id=92f70eacbcca49db85a9d3824786aad6, user_id=0c4db850abc243c887ee39b39baea210, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:41.904658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:41.913126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:41.921308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 924/1851] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:41 2022] POST /identity/v3/projects => generated 355 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:41.922642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.938077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 928/1852] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:41 2022] PUT /identity/v3/projects/92f70eacbcca49db85a9d3824786aad6/users/0c4db850abc243c887ee39b39baea210/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:41.946435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:41.986368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:42.016502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358d7a18-da5c-4ec0-91d4-dd78dbb14759 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.022288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74c8e8e-594b-46a9-9844-3208ebb11700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.023340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74c8e8e-594b-46a9-9844-3208ebb11700 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.023785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74c8e8e-594b-46a9-9844-3208ebb11700 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.024236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74c8e8e-594b-46a9-9844-3208ebb11700 admin admin] PATH_INFO: `/v3/projects/1b907107c8f8461d92b6a2daf658c9ee/users/d7a6bf0045f94f64b4ef8ad9e0164e93/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.026483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74c8e8e-594b-46a9-9844-3208ebb11700 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b907107c8f8461d92b6a2daf658c9ee, user_id=d7a6bf0045f94f64b4ef8ad9e0164e93, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.030250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74c8e8e-594b-46a9-9844-3208ebb11700 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.031754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc96fc5-f2c7-4b65-ab2b-49a64636e225 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.037991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2249eb-3f34-415b-8585-a5b6eb361328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.038694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2249eb-3f34-415b-8585-a5b6eb361328 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.038981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2249eb-3f34-415b-8585-a5b6eb361328 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.039264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2249eb-3f34-415b-8585-a5b6eb361328 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.039963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2249eb-3f34-415b-8585-a5b6eb361328 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.042583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2249eb-3f34-415b-8585-a5b6eb361328 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.043812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 925/1853] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:41 2022] PUT /identity/v3/projects/1b907107c8f8461d92b6a2daf658c9ee/users/d7a6bf0045f94f64b4ef8ad9e0164e93/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:42.045480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e5ca7-3d3f-4df5-a6a4-57ddf8e3e5ab None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.045869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e5ca7-3d3f-4df5-a6a4-57ddf8e3e5ab None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.046242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4e5ca7-3d3f-4df5-a6a4-57ddf8e3e5ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.071000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 929/1854] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:41 2022] POST /identity/v3/users => generated 278 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:42.071000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.071000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.071000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.084351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-3b4e5ca7-3d3f-4df5-a6a4-57ddf8e3e5ab None None] MFA Rules not processed for user `0c4db850abc243c887ee39b39baea210`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:42.085780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3b4e5ca7-3d3f-4df5-a6a4-57ddf8e3e5ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:42.118840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] MFA Rules not processed for user `d7a6bf0045f94f64b4ef8ad9e0164e93`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:42.120140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:42.210374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 926/1855] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:42 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:42.212184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4e5ca7-3d3f-4df5-a6a4-57ddf8e3e5ab None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.236552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 930/1856] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:42 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:42.237714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.241512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:42.286865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4e5ca7-3d3f-4df5-a6a4-57ddf8e3e5ab None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.291864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.292574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.292876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.293143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.293990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.295963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.307940 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 927/1857] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:42 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:42.309173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.311097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:42.314129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dda286-8289-4a22-9597-6dd38c50c1ee None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.319604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.320577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.320882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.321160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.324527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac238b65-4972-4bef-b69a-1d7226e51d5f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.328190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.328918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.329152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.329435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.331370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:42.332278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.333838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.343340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 931/1858] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:42 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:42.344130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.353881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d7f48a-d57e-49b5-a4cb-f2a8f1c5c89a service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.358814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.359089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.359370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.359643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.360435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.361979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.372810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 932/1859] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:42 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:42.379141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.390218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2617f-e8d4-44b9-86ee-e5663f54ecea admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.396749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fa3399-49ee-4dbd-ad70-b45c13a931f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.397862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa3399-49ee-4dbd-ad70-b45c13a931f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.398035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa3399-49ee-4dbd-ad70-b45c13a931f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.398321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fa3399-49ee-4dbd-ad70-b45c13a931f1 admin admin] PATH_INFO: `/v3/projects/9de50fd47c1f4c639b5cf0c1ebb49b96/users/65efedb96a1f4c9d84ec15607b1fff92/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.400950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:42.401062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:42.413749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.415415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.419226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fa3399-49ee-4dbd-ad70-b45c13a931f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9de50fd47c1f4c639b5cf0c1ebb49b96, user_id=65efedb96a1f4c9d84ec15607b1fff92, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.423429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fa3399-49ee-4dbd-ad70-b45c13a931f1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.430783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 928/1860] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:42 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:42.434910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.441062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 933/1861] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:42 2022] PUT /identity/v3/projects/9de50fd47c1f4c639b5cf0c1ebb49b96/users/65efedb96a1f4c9d84ec15607b1fff92/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:42.444081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727cdb12-57cb-4e14-adb6-bb7c6e4b3668 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.452950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa42f945-2dfa-4268-8ee9-e84b0638bedf None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.453344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa42f945-2dfa-4268-8ee9-e84b0638bedf None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.454148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa42f945-2dfa-4268-8ee9-e84b0638bedf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.484285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-aa42f945-2dfa-4268-8ee9-e84b0638bedf None None] MFA Rules not processed for user `65efedb96a1f4c9d84ec15607b1fff92`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:42.485615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-aa42f945-2dfa-4268-8ee9-e84b0638bedf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:42.501946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.502912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.503271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.503730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.506202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:42.572456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 934/1862] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:42 2022] POST /identity/v3/auth/tokens => generated 3222 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:42.575058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa42f945-2dfa-4268-8ee9-e84b0638bedf None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.576418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.578202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.588728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 929/1863] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:42 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:10:42.665377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa42f945-2dfa-4268-8ee9-e84b0638bedf None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.671583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.672071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.672417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.672718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.673446 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.675458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.716223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 935/1864] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:42 2022] POST /identity/v3/projects => generated 357 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:10:42.721570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.736107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b92a8b9-ac11-4d26-bf1c-4249960224cd service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.837948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.838637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.838920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.839212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.840818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.843111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.868674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 930/1865] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:42 2022] POST /identity/v3/users => generated 285 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:42.872596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.883289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d53762-20b1-4dd8-ba86-f0df02978012 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.888360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.888615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.888903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.889180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.889907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.891516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.902393 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 936/1866] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:42.910755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.919154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d3b2f5-8ef9-4101-81e6-ccdea5c44b7d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:42.923816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:42.924749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:42.925036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:42.925314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] PATH_INFO: `/v3/projects/6c4a5c789e384a7e88a70b57bdc3a619/users/531bce66ddc8487cb390029af2ba3587/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:42.941464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c4a5c789e384a7e88a70b57bdc3a619, user_id=531bce66ddc8487cb390029af2ba3587, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:42.944174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:42.958046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 931/1867] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:42 2022] PUT /identity/v3/projects/6c4a5c789e384a7e88a70b57bdc3a619/users/531bce66ddc8487cb390029af2ba3587/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:42.962219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:42.972576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51afe07d-24f1-4ead-9400-28c1155522bd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.003679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:43.016220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:43.022143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:43.036303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.037682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.038021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.038301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.039054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.040692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:43.050562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 937/1868] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:42 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:10:43.054968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:43.064847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0638b2-18d2-4b5a-a90d-93caadeb23e6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.069260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8754ba0b-5943-47ed-967d-ccd214e310c3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.069480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07982b81-2deb-4f22-8488-4d425fe9ef0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.070346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07982b81-2deb-4f22-8488-4d425fe9ef0a admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.070628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07982b81-2deb-4f22-8488-4d425fe9ef0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.070903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07982b81-2deb-4f22-8488-4d425fe9ef0a admin admin] PATH_INFO: `/v3/projects/6c4a5c789e384a7e88a70b57bdc3a619/users/531bce66ddc8487cb390029af2ba3587/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.073222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07982b81-2deb-4f22-8488-4d425fe9ef0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c4a5c789e384a7e88a70b57bdc3a619, user_id=531bce66ddc8487cb390029af2ba3587, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.074317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.075200 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.075489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.075764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07982b81-2deb-4f22-8488-4d425fe9ef0a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:43.075860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] PATH_INFO: `/v3/users/1b47385acd1644aa98f1a30b24a75b59` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.076996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b47385acd1644aa98f1a30b24a75b59)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.078912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:43.086941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 938/1869] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:43 2022] PUT /identity/v3/projects/6c4a5c789e384a7e88a70b57bdc3a619/users/531bce66ddc8487cb390029af2ba3587/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:43.097912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.098950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.099296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.130691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b47385acd1644aa98f1a30b24a75b59'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.132982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] MFA Rules not processed for user `531bce66ddc8487cb390029af2ba3587`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:43.134187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:43.139215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b47385acd1644aa98f1a30b24a75b59'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.139875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b47385acd1644aa98f1a30b24a75b59'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.140645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b47385acd1644aa98f1a30b24a75b59'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.158115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b8ec2317-2f2a-4c97-94a8-4ddac994b66c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b47385acd1644aa98f1a30b24a75b59'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.174483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 932/1870] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:43 2022] DELETE /identity/v3/users/1b47385acd1644aa98f1a30b24a75b59 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:43.178622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.178918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.179196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.198088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:43.199338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:43.215974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 939/1871] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:43 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:10:43.230012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:43.241290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:43.301488 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 933/1872] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:10:43.313722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:43.315567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:43.341057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c244b9dc-5ada-472c-8094-d309696b5fb9 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.387133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:43.418673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.419814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.420123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.420433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.424941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc06d999-19b1-44b5-8286-90494b833ada None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.431105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.431777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.432064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.432340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.444781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.446378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:43.452633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:43.463966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 934/1873] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:43.495458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.497233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:43.509321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 940/1874] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:43 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:43.611633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:43.626022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12abc888-9704-48c0-8640-924fea7c3d54 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.629614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.630267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.631093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.631374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.633183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:43.712255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.713768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:43.723135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 935/1875] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:43 2022] GET /identity/v3/auth/tokens => generated 3222 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:43.785640 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:43.787502 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:43.801542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0623fcb-4eb8-4ea7-a7a1-60516da5caf4 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.806881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.809571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.809813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.810102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] PATH_INFO: `/v3/projects/001d710a50b943808aa74cb9a17b25a7` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.811414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] RBAC: Authorizing `identity:delete_project(project_id=001d710a50b943808aa74cb9a17b25a7)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.813534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:43.889246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '001d710a50b943808aa74cb9a17b25a7', 'request_id': 'req-48b3e00a-78ec-4f28-b627-9beaf027d3da'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.894799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '001d710a50b943808aa74cb9a17b25a7', 'request_id': 'req-48b3e00a-78ec-4f28-b627-9beaf027d3da'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.902721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '001d710a50b943808aa74cb9a17b25a7', 'request_id': 'req-48b3e00a-78ec-4f28-b627-9beaf027d3da'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.914093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] The token cache is being invalidate because project 001d710a50b943808aa74cb9a17b25a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:43.914382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.915005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:43.916863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 941/1876] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:43 2022] DELETE /identity/v3/projects/001d710a50b943808aa74cb9a17b25a7 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:43.920968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:43.934692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488c207b-76de-43da-a25a-1550c73592a4 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:43.992286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:43.993198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:43.993482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:43.993758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] PATH_INFO: `/v3/users/0c4db850abc243c887ee39b39baea210` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:43.994935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c4db850abc243c887ee39b39baea210)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:43.996925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:44.041406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c4db850abc243c887ee39b39baea210'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.053980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c4db850abc243c887ee39b39baea210'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.054706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c4db850abc243c887ee39b39baea210'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.055306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c4db850abc243c887ee39b39baea210'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.067423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-986e62e4-aa8f-407b-9563-cc34fe334781 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c4db850abc243c887ee39b39baea210'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.069753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:44.083246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 936/1877] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:43 2022] DELETE /identity/v3/users/0c4db850abc243c887ee39b39baea210 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:44.353804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:44.354943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:44.368861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3e00a-78ec-4f28-b627-9beaf027d3da admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:44.435685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:44.436772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.437153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.437505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] PATH_INFO: `/v3/projects/92f70eacbcca49db85a9d3824786aad6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.438824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] RBAC: Authorizing `identity:delete_project(project_id=92f70eacbcca49db85a9d3824786aad6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:44.440883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:44.508576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:44.511817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '92f70eacbcca49db85a9d3824786aad6', 'request_id': 'req-e840c5a5-5fdc-4243-a795-dede14a1845a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.518912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '92f70eacbcca49db85a9d3824786aad6', 'request_id': 'req-e840c5a5-5fdc-4243-a795-dede14a1845a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.527615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '92f70eacbcca49db85a9d3824786aad6', 'request_id': 'req-e840c5a5-5fdc-4243-a795-dede14a1845a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.539466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] The token cache is being invalidate because project 92f70eacbcca49db85a9d3824786aad6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:44.539757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.540421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e840c5a5-5fdc-4243-a795-dede14a1845a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:44.542105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 942/1878] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:44 2022] DELETE /identity/v3/projects/92f70eacbcca49db85a9d3824786aad6 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:44.547212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b912e982-3213-4639-afdc-eb87dbb328b1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.547612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b912e982-3213-4639-afdc-eb87dbb328b1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.547986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b912e982-3213-4639-afdc-eb87dbb328b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.570558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b912e982-3213-4639-afdc-eb87dbb328b1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:44.572247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b912e982-3213-4639-afdc-eb87dbb328b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:44.634414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 937/1879] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:44.642183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9395c647-56b2-43b2-a80d-6b3ad7d09f83 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.642624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9395c647-56b2-43b2-a80d-6b3ad7d09f83 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.642914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9395c647-56b2-43b2-a80d-6b3ad7d09f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.669473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9395c647-56b2-43b2-a80d-6b3ad7d09f83 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:44.670706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9395c647-56b2-43b2-a80d-6b3ad7d09f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:44.683099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 943/1880] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:44.687431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b912e982-3213-4639-afdc-eb87dbb328b1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:44.698147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b912e982-3213-4639-afdc-eb87dbb328b1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:44.702793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:44.703706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.703996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.705715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.706499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:44.706816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:44.708805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:44.722643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 938/1881] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:44.727749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9395c647-56b2-43b2-a80d-6b3ad7d09f83 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:44.739655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9395c647-56b2-43b2-a80d-6b3ad7d09f83 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:44.745354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:44.746029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.746332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.746830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.747639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:44.749606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:44.781417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 944/1882] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:44 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:44.782686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:44.793119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc41fcc-0b2d-4706-94b5-966911752e2d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:44.852644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:44.853326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.853946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.853946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.854553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:44.856652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:44.876151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 939/1883] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:44 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:44.880586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:44.894346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76395648-8a83-4520-84b6-8b1341a913bd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:44.899281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:44.900147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.902098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.902098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.902098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:44.903544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:44.921058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 945/1884] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:44 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:44.921499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:44.936007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764e0043-7e41-44c9-9ea0-d3c46bbf0c81 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:44.941482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769faa81-4b8b-4f26-bdbe-fc68b130fca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:44.942402 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769faa81-4b8b-4f26-bdbe-fc68b130fca5 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.942758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769faa81-4b8b-4f26-bdbe-fc68b130fca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.943146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769faa81-4b8b-4f26-bdbe-fc68b130fca5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.943884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769faa81-4b8b-4f26-bdbe-fc68b130fca5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:44.945619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769faa81-4b8b-4f26-bdbe-fc68b130fca5 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:44.960491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 940/1885] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:44 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:44.961654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:44.971992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d283ac-92b9-41ca-b3fe-ddff8ed94f4c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:44.977233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:44.978123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:44.978408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:44.978686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] PATH_INFO: `/v3/projects/ed027593e6c046bab4ec67f43aa686d8/users/ac6cde27ebdf4d14a65196a65d62ef1c/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:44.993818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed027593e6c046bab4ec67f43aa686d8, user_id=ac6cde27ebdf4d14a65196a65d62ef1c, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:44.996539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:45.009481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 946/1886] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:44 2022] PUT /identity/v3/projects/ed027593e6c046bab4ec67f43aa686d8/users/ac6cde27ebdf4d14a65196a65d62ef1c/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:45.019974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a91a0-520d-40a7-ae5d-f9ad8bc7f08b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:45.020270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a91a0-520d-40a7-ae5d-f9ad8bc7f08b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:45.020585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02a91a0-520d-40a7-ae5d-f9ad8bc7f08b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:45.051005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b02a91a0-520d-40a7-ae5d-f9ad8bc7f08b None None] MFA Rules not processed for user `ac6cde27ebdf4d14a65196a65d62ef1c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:45.052215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b02a91a0-520d-40a7-ae5d-f9ad8bc7f08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:45.096000 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:45.117737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 941/1887] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:45 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:45.128756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:45.130448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:45.186401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f814d442-90b2-4f41-8339-1ab7fe287ef6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:45.190218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd595577-5d57-437d-870a-d7f6ec0de4c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:45.190808 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd595577-5d57-437d-870a-d7f6ec0de4c6 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:45.191094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd595577-5d57-437d-870a-d7f6ec0de4c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:45.191374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd595577-5d57-437d-870a-d7f6ec0de4c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:45.200041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd595577-5d57-437d-870a-d7f6ec0de4c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:45.201501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd595577-5d57-437d-870a-d7f6ec0de4c6 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:45.210453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 947/1888] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:45 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:45.229334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02a91a0-520d-40a7-ae5d-f9ad8bc7f08b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:45.291434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02a91a0-520d-40a7-ae5d-f9ad8bc7f08b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:45.296106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:45.296997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:45.297276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:45.297547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] PATH_INFO: `/v3/users/ac6cde27ebdf4d14a65196a65d62ef1c` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:45.298730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac6cde27ebdf4d14a65196a65d62ef1c)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:45.300674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:45.345581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac6cde27ebdf4d14a65196a65d62ef1c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:45.353320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac6cde27ebdf4d14a65196a65d62ef1c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:45.354242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac6cde27ebdf4d14a65196a65d62ef1c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:45.354884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac6cde27ebdf4d14a65196a65d62ef1c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:45.367264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac6cde27ebdf4d14a65196a65d62ef1c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:45.385536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 942/1889] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:10:45 2022] DELETE /identity/v3/users/ac6cde27ebdf4d14a65196a65d62ef1c => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:45.391542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:45.392541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:45.394030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:45.413437 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:45.414665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:45.487731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 948/1890] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:45.499611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:45.501712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:45.561675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464f8549-0436-46b3-8533-a964f91cf41f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:45.565294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c93165-8d6d-47d2-8b7f-e66a1ad4aa6d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:45.565945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c93165-8d6d-47d2-8b7f-e66a1ad4aa6d service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:45.566228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c93165-8d6d-47d2-8b7f-e66a1ad4aa6d service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:45.566503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c93165-8d6d-47d2-8b7f-e66a1ad4aa6d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:45.575575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c93165-8d6d-47d2-8b7f-e66a1ad4aa6d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:45.577057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c93165-8d6d-47d2-8b7f-e66a1ad4aa6d service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:45.586808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 943/1891] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:46.219676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.221369 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:46.244734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b33253-0b44-446e-9e2c-451dbb906c1d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.259094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.259991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.260274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.260597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] PATH_INFO: `/v3/projects/ed027593e6c046bab4ec67f43aa686d8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.266163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed027593e6c046bab4ec67f43aa686d8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.266163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.275497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:DELETE=4,keystone:SELECT=93,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:46.319595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed027593e6c046bab4ec67f43aa686d8', 'request_id': 'req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:46.325491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed027593e6c046bab4ec67f43aa686d8', 'request_id': 'req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:46.334447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed027593e6c046bab4ec67f43aa686d8', 'request_id': 'req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:46.346021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] The token cache is being invalidate because project ed027593e6c046bab4ec67f43aa686d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:10:46.346301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:46.347243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5a40ef99-e3b1-4eae-a272-f78e70af6ce2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:10:46.348952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 949/1892] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:10:46 2022] DELETE /identity/v3/projects/ed027593e6c046bab4ec67f43aa686d8 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:10:46.354943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857a94c5-e0e7-485f-a132-40fc91e50ea3 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.355922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857a94c5-e0e7-485f-a132-40fc91e50ea3 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.356068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857a94c5-e0e7-485f-a132-40fc91e50ea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.375369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-857a94c5-e0e7-485f-a132-40fc91e50ea3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:46.376656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-857a94c5-e0e7-485f-a132-40fc91e50ea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:46.382646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:46.452055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 944/1893] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:46.455688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2646e6-261a-4457-a437-e497f846bbed None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.456130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2646e6-261a-4457-a437-e497f846bbed None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.456480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2646e6-261a-4457-a437-e497f846bbed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.481100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2e2646e6-261a-4457-a437-e497f846bbed None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:46.482345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2e2646e6-261a-4457-a437-e497f846bbed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:46.491061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 950/1894] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:46.495249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857a94c5-e0e7-485f-a132-40fc91e50ea3 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.506337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857a94c5-e0e7-485f-a132-40fc91e50ea3 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.511198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.511798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.512080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.512358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.513117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:10:46.513424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.515075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.527413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 945/1895] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:10:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:10:46.531764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2646e6-261a-4457-a437-e497f846bbed None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.542076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2646e6-261a-4457-a437-e497f846bbed None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.546924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.547608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.547893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.548169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.548908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.550854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.577500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 951/1896] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:10:46 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:46.581512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.591535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d659824d-1e4c-44c7-95cb-078112803661 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.650113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.650786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.651074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.651352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.652035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.653990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.674326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 946/1897] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:10:46 2022] POST /identity/v3/users => generated 285 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:10:46.679003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.690667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4537104-6c74-4c06-aaad-70f33f8f9ea9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.695450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.696119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.696434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.696717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.697714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.699356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.709792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 952/1898] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:46.713700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.729121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e770299e-cb1c-45df-a9fd-28d8791f5ecb admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.733860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9963c9-3045-4b3d-934c-7424d8748c8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.734530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9963c9-3045-4b3d-934c-7424d8748c8f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.734815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9963c9-3045-4b3d-934c-7424d8748c8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.735092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9963c9-3045-4b3d-934c-7424d8748c8f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.735823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9963c9-3045-4b3d-934c-7424d8748c8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.737479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9963c9-3045-4b3d-934c-7424d8748c8f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.748411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 947/1899] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:10:46 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:10:46.752587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.763385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276309b9-695e-49a2-ac8e-b44ffcee1175 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.768043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.768958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.769271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.769549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] PATH_INFO: `/v3/projects/c48f677354594fdc839367fff8b3dc29/users/d2d25be9c4674aa09b2975f1c6bc8c52/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.785459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c48f677354594fdc839367fff8b3dc29, user_id=d2d25be9c4674aa09b2975f1c6bc8c52, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.788852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.802682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 953/1900] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:10:46 2022] PUT /identity/v3/projects/c48f677354594fdc839367fff8b3dc29/users/d2d25be9c4674aa09b2975f1c6bc8c52/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:10:46.807572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7890d-4671-468a-ba00-a083c919223b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.807889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7890d-4671-468a-ba00-a083c919223b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.808149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f7890d-4671-468a-ba00-a083c919223b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.826472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-67f7890d-4671-468a-ba00-a083c919223b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:46.827598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-67f7890d-4671-468a-ba00-a083c919223b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:46.891185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 948/1901] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:46.902470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:46.904420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:46.966331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5c9d0a-acce-402a-ba6b-607390f37c39 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:46.970006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:46.970661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:46.970947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:46.971224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:46.981336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:46.982756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:46.992341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 954/1902] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:47.894756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177b92f3-b733-451f-800b-418f7f6cd844 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:47.895240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177b92f3-b733-451f-800b-418f7f6cd844 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:47.895343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177b92f3-b733-451f-800b-418f7f6cd844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:47.921972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-177b92f3-b733-451f-800b-418f7f6cd844 None None] MFA Rules not processed for user `d2d25be9c4674aa09b2975f1c6bc8c52`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:47.923195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-177b92f3-b733-451f-800b-418f7f6cd844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:47.928488 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:47.992463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 949/1903] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:10:47 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:48.003299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:48.004778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:48.044968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:10:48.063230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241ad478-6d2c-4da1-b6a3-68256607987e service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:48.066786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad4d605-a8a9-4dd6-a702-aa4e32880f34 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:48.067434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad4d605-a8a9-4dd6-a702-aa4e32880f34 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:48.067715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad4d605-a8a9-4dd6-a702-aa4e32880f34 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:48.068045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad4d605-a8a9-4dd6-a702-aa4e32880f34 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:48.077443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad4d605-a8a9-4dd6-a702-aa4e32880f34 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:48.078869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad4d605-a8a9-4dd6-a702-aa4e32880f34 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:48.087645 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 955/1904] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:48 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:56.268970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea95219a-ee1d-4e4b-930d-44124669e20f None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:56.269489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea95219a-ee1d-4e4b-930d-44124669e20f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:56.270064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea95219a-ee1d-4e4b-930d-44124669e20f None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:56.271493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 950/1905] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:10:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:10:56.282163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:56.282605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:56.282919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:56.304769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:10:56.306003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:56.386499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 956/1906] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:10:56 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:10:56.400223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea95219a-ee1d-4e4b-930d-44124669e20f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:56.401918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ea95219a-ee1d-4e4b-930d-44124669e20f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:56.418996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea95219a-ee1d-4e4b-930d-44124669e20f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:56.424446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f5c61-11e3-489f-bb4f-d9b721afb84f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:56.425105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f5c61-11e3-489f-bb4f-d9b721afb84f service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:56.425390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f5c61-11e3-489f-bb4f-d9b721afb84f service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:56.425667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f5c61-11e3-489f-bb4f-d9b721afb84f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:56.434697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260f5c61-11e3-489f-bb4f-d9b721afb84f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:56.436217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260f5c61-11e3-489f-bb4f-d9b721afb84f service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:56.446989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 951/1907] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:10:57.750585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:10:57.752085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:57.763105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f00d18-ed2a-4f02-8c12-6869a7d474b0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:10:57.766663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:10:57.767293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:10:57.767575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:10:57.767852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:10:57.769890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:10:57.838222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:10:57.839696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:10:57.849267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 957/1908] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:10:57 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:03.092403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db2f49-7aee-4ef5-8ecc-1124e91addca None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:03.092896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db2f49-7aee-4ef5-8ecc-1124e91addca None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:03.093081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88db2f49-7aee-4ef5-8ecc-1124e91addca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:03.110638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-88db2f49-7aee-4ef5-8ecc-1124e91addca None None] MFA Rules not processed for user `d2d25be9c4674aa09b2975f1c6bc8c52`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:03.111774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-88db2f49-7aee-4ef5-8ecc-1124e91addca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:03.118870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 952/1909] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:03 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:07.845035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=71 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:09.190444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:09.191963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:09.206336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6332430-d29f-4708-8ed7-f33b85dd603e service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:09.211038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:09.212715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:09.213013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:09.213292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] PATH_INFO: `/v3/users/4f5225202d2d408abb981f1fe28fa4fe` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:09.214510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f5225202d2d408abb981f1fe28fa4fe)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:09.216478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:09.266425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f5225202d2d408abb981f1fe28fa4fe'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.267070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f5225202d2d408abb981f1fe28fa4fe'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.274715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f5225202d2d408abb981f1fe28fa4fe'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.275416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f5225202d2d408abb981f1fe28fa4fe'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.284999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f5225202d2d408abb981f1fe28fa4fe'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.308896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 958/1910] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:09 2022] DELETE /identity/v3/users/4f5225202d2d408abb981f1fe28fa4fe => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:09.313176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:09.313644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:09.313757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:09.337485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:09.338706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:09.396432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 953/1911] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:09.406128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:09.407545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:09.465630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd89766-3666-4e58-843c-581da86d9417 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:09.469580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:09.470417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:09.470506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:09.470756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:09.480606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:09.482036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:09.498255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 959/1912] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:09.822845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:09.824591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:09.847538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43f9dba-4079-4992-a6db-0ee16fc5c16e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:09.854175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:09.856080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:09.856968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:09.857701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] PATH_INFO: `/v3/projects/a7deda2682de45eebe55f0050d5b2014` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:09.859434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7deda2682de45eebe55f0050d5b2014)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:09.861696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:09.950186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7deda2682de45eebe55f0050d5b2014', 'request_id': 'req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.955838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7deda2682de45eebe55f0050d5b2014', 'request_id': 'req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.968964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7deda2682de45eebe55f0050d5b2014', 'request_id': 'req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.983114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] The token cache is being invalidate because project a7deda2682de45eebe55f0050d5b2014 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:09.983403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.984213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:09.986133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 954/1913] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:09 2022] DELETE /identity/v3/projects/a7deda2682de45eebe55f0050d5b2014 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:09.993966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:10.027942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049f0506-ca55-4c8e-b68f-8a4c8bd34fb4 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:10.088477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:10.089417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:10.089704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:10.089984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] PATH_INFO: `/v3/users/4bf26700c704482eaeaa18587074f50d` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:10.091220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4bf26700c704482eaeaa18587074f50d)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:10.093213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:10.109186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:10.149148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4bf26700c704482eaeaa18587074f50d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.149266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4bf26700c704482eaeaa18587074f50d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.157508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4bf26700c704482eaeaa18587074f50d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.158299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4bf26700c704482eaeaa18587074f50d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.168055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7efeb2dc-36a5-4a5e-8885-32a8ac14ba17 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4bf26700c704482eaeaa18587074f50d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.189383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 960/1914] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:09 2022] DELETE /identity/v3/users/4bf26700c704482eaeaa18587074f50d => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:10.568285 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:10.570346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:10.591476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a5d5f-6fc6-42c3-b096-0b1b7768483a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:10.599513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:10.658608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:10.659480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:10.659764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:10.660046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] PATH_INFO: `/v3/projects/bcae2d3aae7b47e08f985c56c18dda7a` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:10.661262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcae2d3aae7b47e08f985c56c18dda7a)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:10.663212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:10.708334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcae2d3aae7b47e08f985c56c18dda7a', 'request_id': 'req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.714007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcae2d3aae7b47e08f985c56c18dda7a', 'request_id': 'req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.720139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcae2d3aae7b47e08f985c56c18dda7a', 'request_id': 'req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.733218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] The token cache is being invalidate because project bcae2d3aae7b47e08f985c56c18dda7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:10.733504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.734135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc4aadc7-3823-4998-9dbd-32f2b6ed85a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:10.735885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 955/1915] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:10 2022] DELETE /identity/v3/projects/bcae2d3aae7b47e08f985c56c18dda7a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:10.740667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b2acb-9b03-4015-8498-b64ef91026cc None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:10.741087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b2acb-9b03-4015-8498-b64ef91026cc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:10.741324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b2acb-9b03-4015-8498-b64ef91026cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:10.759921 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0b0b2acb-9b03-4015-8498-b64ef91026cc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:10.761154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0b0b2acb-9b03-4015-8498-b64ef91026cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:10.824428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 961/1916] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:10.828227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5842ab-8ff3-4edd-9898-405b7d2c4c66 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:10.828600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5842ab-8ff3-4edd-9898-405b7d2c4c66 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:10.828881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5842ab-8ff3-4edd-9898-405b7d2c4c66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:10.847577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1a5842ab-8ff3-4edd-9898-405b7d2c4c66 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:10.849022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1a5842ab-8ff3-4edd-9898-405b7d2c4c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:10.857624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 956/1917] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:11:10.861792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0b2acb-9b03-4015-8498-b64ef91026cc None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:10.871543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0b2acb-9b03-4015-8498-b64ef91026cc None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:10.876198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:10.876893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:10.877184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:10.877462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:10.878222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:11:10.878528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:10.880144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:10.892901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 962/1918] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:11:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:11:10.897498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5842ab-8ff3-4edd-9898-405b7d2c4c66 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:10.907725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5842ab-8ff3-4edd-9898-405b7d2c4c66 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:10.912271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:10.912961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:10.913245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:10.913522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:10.914276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:10.916771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:10.949738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 957/1919] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:10 2022] POST /identity/v3/projects => generated 369 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:11:10.953885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:10.964583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e5c245c-9d38-4faf-86e5-4c5bdc46b74c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.025754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.026540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.026834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.027117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.027861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.030257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.056269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 963/1920] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:10 2022] POST /identity/v3/users => generated 285 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:11.060448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.073477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61077cd-f303-4ae4-8ca5-94f2908fb917 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.079882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.080679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.081050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.081418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.085821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.085821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.098867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 958/1921] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:11:11.103495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.126970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258f4d44-bf8a-4c87-a354-2566936faadd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.133084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61b3c73-77e8-4049-9c5e-f39ee26acb06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.134652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61b3c73-77e8-4049-9c5e-f39ee26acb06 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.135354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61b3c73-77e8-4049-9c5e-f39ee26acb06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.135811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61b3c73-77e8-4049-9c5e-f39ee26acb06 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.136794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61b3c73-77e8-4049-9c5e-f39ee26acb06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.138728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61b3c73-77e8-4049-9c5e-f39ee26acb06 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.147106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:11.154441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 964/1922] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:11:11.162037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.180901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bef9da3-87df-43e7-a951-66e42549588f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.186037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.186933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.187221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.187503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] PATH_INFO: `/v3/projects/e792452a73034ec6b8e9749a0515a9fd/users/faf826da24384204b5e6362ae410c9f5/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.208874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e792452a73034ec6b8e9749a0515a9fd, user_id=faf826da24384204b5e6362ae410c9f5, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.213048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.231147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 959/1923] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:11 2022] PUT /identity/v3/projects/e792452a73034ec6b8e9749a0515a9fd/users/faf826da24384204b5e6362ae410c9f5/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:11.243407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056b9102-c398-4505-889e-c6b4a1f72de2 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.243798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056b9102-c398-4505-889e-c6b4a1f72de2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.244080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056b9102-c398-4505-889e-c6b4a1f72de2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.277439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-056b9102-c398-4505-889e-c6b4a1f72de2 None None] MFA Rules not processed for user `faf826da24384204b5e6362ae410c9f5`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:11.278823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-056b9102-c398-4505-889e-c6b4a1f72de2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:11.372843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 965/1924] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:11 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:11:11.377087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.434485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:11.462976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfd384-6add-446a-bfae-88fccaae3888 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.468217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.469111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.469744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.469744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.470371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.472300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.502657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 960/1925] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:11 2022] POST /identity/v3/projects => generated 367 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:11.506674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056b9102-c398-4505-889e-c6b4a1f72de2 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.522259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056b9102-c398-4505-889e-c6b4a1f72de2 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.582725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.583438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.583726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.584006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.584773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.586705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.610237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 966/1926] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:11 2022] POST /identity/v3/users => generated 290 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:11.611715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.622597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7062281-976e-4252-9b16-6b108f37ad12 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.627305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.627976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.628262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.628572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.629297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.630946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.642571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 961/1927] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:11.646922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.657338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0720c21-8b38-4880-827f-6902e2051913 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.662189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.663091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.663380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.663658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] PATH_INFO: `/v3/projects/c78bc139dd2f42358d581430d6d9351c/users/db8306686f424341ab2b4f232b717a88/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.679822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c78bc139dd2f42358d581430d6d9351c, user_id=db8306686f424341ab2b4f232b717a88, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.683089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.699453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 967/1928] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:11 2022] PUT /identity/v3/projects/c78bc139dd2f42358d581430d6d9351c/users/db8306686f424341ab2b4f232b717a88/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:11.703217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.713205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470536c2-2123-48f1-9db1-c26658e81dde admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.776572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7212200-ef00-4158-ae0f-9e0244acd62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.777244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7212200-ef00-4158-ae0f-9e0244acd62f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.777529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7212200-ef00-4158-ae0f-9e0244acd62f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.777808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7212200-ef00-4158-ae0f-9e0244acd62f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.778535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7212200-ef00-4158-ae0f-9e0244acd62f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.780127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7212200-ef00-4158-ae0f-9e0244acd62f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.790074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 962/1929] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:11 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:11:11.793957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.804221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c69f63-b9b9-46d2-b3f2-51c0b5e849f3 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:11.809373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:11.810255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.810539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.810814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] PATH_INFO: `/v3/projects/c78bc139dd2f42358d581430d6d9351c/users/db8306686f424341ab2b4f232b717a88/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.812848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c78bc139dd2f42358d581430d6d9351c, user_id=db8306686f424341ab2b4f232b717a88, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:11.815495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:11.826805 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 968/1930] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:11 2022] PUT /identity/v3/projects/c78bc139dd2f42358d581430d6d9351c/users/db8306686f424341ab2b4f232b717a88/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:11.836919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:11.837212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:11.837489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:11.863548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] MFA Rules not processed for user `db8306686f424341ab2b4f232b717a88`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:11.864778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:11.934717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 963/1931] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:11 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:11.946038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:11.947719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:11.994250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:12.007604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c13fd1-a9fd-4583-81d9-fc1fd4a053d9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:12.011308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:12.011954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:12.012240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:12.012554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:12.021630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:12.023057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:12.032871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 969/1932] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:11 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:12.076269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:12.077786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:12.090006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf0661d-7f9a-4369-a878-31de2aa9438d None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:12.094162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f195043c-516d-42c0-8c66-531d34e2f119 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:12.094809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f195043c-516d-42c0-8c66-531d34e2f119 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:12.095135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f195043c-516d-42c0-8c66-531d34e2f119 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:12.095351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f195043c-516d-42c0-8c66-531d34e2f119 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:12.127620 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:12.156998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f195043c-516d-42c0-8c66-531d34e2f119 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:12.158438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f195043c-516d-42c0-8c66-531d34e2f119 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:12.168219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 964/1933] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:12 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:15.599343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:15.665043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41ea9eb-7db7-4d03-b331-e761cc7978d9 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:15.669826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:15.670708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:15.671008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:15.671271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] PATH_INFO: `/v3/users/faf826da24384204b5e6362ae410c9f5` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:15.672473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=faf826da24384204b5e6362ae410c9f5)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:15.674406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:15.722230 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'faf826da24384204b5e6362ae410c9f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:15.722992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'faf826da24384204b5e6362ae410c9f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:15.730651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'faf826da24384204b5e6362ae410c9f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:15.731389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'faf826da24384204b5e6362ae410c9f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:15.740536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'faf826da24384204b5e6362ae410c9f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:15.759600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 970/1934] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:15 2022] DELETE /identity/v3/users/faf826da24384204b5e6362ae410c9f5 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:15.763853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:15.764369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:15.764679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:15.782232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:15.783545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:15.853706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 965/1935] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:15.863203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:15.864656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:15.924718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5aad7a-094c-45f5-9182-02ca880daaf4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:15.929078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:15.929785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:15.930077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:15.930375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:15.942255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:15.943759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:15.954289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 971/1936] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:16.300652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:16.302237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:16.320322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff07956-9c0c-4dcb-a6df-a92421b3d039 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:16.327969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:16.329024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:16.329309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:16.329587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] PATH_INFO: `/v3/projects/e792452a73034ec6b8e9749a0515a9fd` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:16.330796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e792452a73034ec6b8e9749a0515a9fd)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:16.332921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:16.408458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e792452a73034ec6b8e9749a0515a9fd', 'request_id': 'req-71c5b660-d483-460d-be15-61ba411110e6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.420550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e792452a73034ec6b8e9749a0515a9fd', 'request_id': 'req-71c5b660-d483-460d-be15-61ba411110e6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.429981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e792452a73034ec6b8e9749a0515a9fd', 'request_id': 'req-71c5b660-d483-460d-be15-61ba411110e6'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.464203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] The token cache is being invalidate because project e792452a73034ec6b8e9749a0515a9fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:16.464650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.465437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.468835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 966/1937] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:16 2022] DELETE /identity/v3/projects/e792452a73034ec6b8e9749a0515a9fd => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:16.475384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:16.487715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6797c7d7-1889-4088-89af-8c82f07ce0df service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:16.499662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:16.622069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:16.624310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:16.624775 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:16.625567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] PATH_INFO: `/v3/users/db8306686f424341ab2b4f232b717a88` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:16.627772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=db8306686f424341ab2b4f232b717a88)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:16.631077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:16.717125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db8306686f424341ab2b4f232b717a88'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.717877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db8306686f424341ab2b4f232b717a88'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.726754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db8306686f424341ab2b4f232b717a88'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.727633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db8306686f424341ab2b4f232b717a88'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.739713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-82d4e3b6-b8f6-4b7e-8276-07c1110e4f72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db8306686f424341ab2b4f232b717a88'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:16.779289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 972/1938] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:16 2022] DELETE /identity/v3/users/db8306686f424341ab2b4f232b717a88 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:17.396836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:17.399301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:17.419356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c5b660-d483-460d-be15-61ba411110e6 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:17.483579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:17.496492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:17.497535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.498045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.498328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] PATH_INFO: `/v3/projects/c78bc139dd2f42358d581430d6d9351c` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.499537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c78bc139dd2f42358d581430d6d9351c)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:17.501487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:17.546127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c78bc139dd2f42358d581430d6d9351c', 'request_id': 'req-5791b592-e514-42da-80af-9a4455b1ba22'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:17.551495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c78bc139dd2f42358d581430d6d9351c', 'request_id': 'req-5791b592-e514-42da-80af-9a4455b1ba22'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:17.557996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c78bc139dd2f42358d581430d6d9351c', 'request_id': 'req-5791b592-e514-42da-80af-9a4455b1ba22'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:17.569577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] The token cache is being invalidate because project c78bc139dd2f42358d581430d6d9351c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:17.569865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:17.570475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5791b592-e514-42da-80af-9a4455b1ba22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:17.572687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 967/1939] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:17 2022] DELETE /identity/v3/projects/c78bc139dd2f42358d581430d6d9351c => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:17.578073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9de6201-7f9b-49c0-9839-d863036b67a7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.578373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9de6201-7f9b-49c0-9839-d863036b67a7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.578654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9de6201-7f9b-49c0-9839-d863036b67a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.596815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a9de6201-7f9b-49c0-9839-d863036b67a7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:17.598016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a9de6201-7f9b-49c0-9839-d863036b67a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:17.672043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 973/1940] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:17.684688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b1996b-2c88-4f59-97fc-1da58c69db3e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.684981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b1996b-2c88-4f59-97fc-1da58c69db3e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.685262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b1996b-2c88-4f59-97fc-1da58c69db3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.707070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e2b1996b-2c88-4f59-97fc-1da58c69db3e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:17.708291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e2b1996b-2c88-4f59-97fc-1da58c69db3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:17.716985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 968/1941] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:11:17.721299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9de6201-7f9b-49c0-9839-d863036b67a7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:17.731648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9de6201-7f9b-49c0-9839-d863036b67a7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:17.736508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:17.737378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.737666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.737952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.738696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:11:17.739009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:17.741078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:17.762337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 974/1942] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:11:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:11:17.766808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b1996b-2c88-4f59-97fc-1da58c69db3e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:17.777439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b1996b-2c88-4f59-97fc-1da58c69db3e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:17.782209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:17.782832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.783119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.783395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.784263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:17.786111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:17.814612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 969/1943] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:17 2022] POST /identity/v3/projects => generated 407 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:11:17.818707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:17.829563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d0c85c-c44f-46ec-94e2-b4e96b429b3c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:17.838478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:17.893609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:17.894203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.894489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.894767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.895448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:17.897416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:17.917402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 975/1944] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:17 2022] POST /identity/v3/users => generated 304 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:17.921423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:17.932633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a867243-73c1-47a4-bea7-6e54198108d0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:17.937241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:17.937909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.938196 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.938478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.939204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:17.940852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:17.951563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 970/1945] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:17.955468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:17.965815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33515c46-d14e-4d1c-bf29-f73d7164c6ea admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:17.970379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4cd33f-2226-4b87-969e-10cdaa2f83c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:17.971036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4cd33f-2226-4b87-969e-10cdaa2f83c6 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:17.971320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4cd33f-2226-4b87-969e-10cdaa2f83c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:17.971598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4cd33f-2226-4b87-969e-10cdaa2f83c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:17.972324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4cd33f-2226-4b87-969e-10cdaa2f83c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:17.973993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4cd33f-2226-4b87-969e-10cdaa2f83c6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:17.984756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 976/1946] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:17.988814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:17.999486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c161b6a-d38e-4a96-bc89-50d27c230c17 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.004215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.005182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.005547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.005888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] PATH_INFO: `/v3/projects/c783e271fb47447d9a8e63c8c3e827bf/users/ad941a8952cb489fb18a2bebdc9319d2/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.021044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c783e271fb47447d9a8e63c8c3e827bf, user_id=ad941a8952cb489fb18a2bebdc9319d2, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.023784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.036970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 971/1947] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:17 2022] PUT /identity/v3/projects/c783e271fb47447d9a8e63c8c3e827bf/users/ad941a8952cb489fb18a2bebdc9319d2/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:18.047936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962d4cc3-d1b0-426d-826f-dad4887ca9ea None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.048229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962d4cc3-d1b0-426d-826f-dad4887ca9ea None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.048536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962d4cc3-d1b0-426d-826f-dad4887ca9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.076653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-962d4cc3-d1b0-426d-826f-dad4887ca9ea None None] MFA Rules not processed for user `ad941a8952cb489fb18a2bebdc9319d2`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:18.077858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-962d4cc3-d1b0-426d-826f-dad4887ca9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:18.142615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 977/1948] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:18 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:18.149068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.211967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4cd5ec-9dd0-46e8-9e00-f0b92e200228 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.216686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.217353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.217636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.217916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.218617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.220555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.249782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 972/1949] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:18 2022] POST /identity/v3/projects => generated 407 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:18.251249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962d4cc3-d1b0-426d-826f-dad4887ca9ea None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.262203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962d4cc3-d1b0-426d-826f-dad4887ca9ea None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.322161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.322840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.323129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.323411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.324105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.326386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.347571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 978/1950] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:18 2022] POST /identity/v3/users => generated 310 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:11:18.350420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.357556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:18.361255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa131a0a-82e1-4651-b0f0-f7ce0de1842e admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.367481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.368646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.368646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.369090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.370060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.371990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.383015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 973/1951] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:18.387169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.397231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a52af9d-9adb-441f-a8c9-f4346943e951 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.402687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.403682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.403962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.404237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] PATH_INFO: `/v3/projects/e0f93936e3444cab851397ace68a6341/users/8dc8994198ed429c85b7e32b60a5063a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.414602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:18.425443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0f93936e3444cab851397ace68a6341, user_id=8dc8994198ed429c85b7e32b60a5063a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.428737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.445761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 979/1952] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:18 2022] PUT /identity/v3/projects/e0f93936e3444cab851397ace68a6341/users/8dc8994198ed429c85b7e32b60a5063a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:11:18.450117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.462865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1591846a-7648-4d6d-a5c4-9f46de114583 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.518136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a065ffe-cf3b-4822-8a39-c6d98b0be1be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.518804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a065ffe-cf3b-4822-8a39-c6d98b0be1be admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.519087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a065ffe-cf3b-4822-8a39-c6d98b0be1be admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.519363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a065ffe-cf3b-4822-8a39-c6d98b0be1be admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.520085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a065ffe-cf3b-4822-8a39-c6d98b0be1be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.521709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a065ffe-cf3b-4822-8a39-c6d98b0be1be admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.531834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 974/1953] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:18 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:18.535983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.546975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fed0e0-58c6-4246-9b32-f34e4468bf12 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.551558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.552466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.552797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.553063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] PATH_INFO: `/v3/projects/e0f93936e3444cab851397ace68a6341/users/8dc8994198ed429c85b7e32b60a5063a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.555060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0f93936e3444cab851397ace68a6341, user_id=8dc8994198ed429c85b7e32b60a5063a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.557765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.568716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 980/1954] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:18 2022] PUT /identity/v3/projects/e0f93936e3444cab851397ace68a6341/users/8dc8994198ed429c85b7e32b60a5063a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:18.579220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.579519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.579804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.607035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] MFA Rules not processed for user `8dc8994198ed429c85b7e32b60a5063a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:18.608289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:18.675224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 975/1955] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:18 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:18.685642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.687352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:18.744155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634afbb3-28f7-4e71-a47c-634cb626e2c9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:18.747968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:18.748658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:18.748944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:18.749224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:18.758226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:18.760179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:18.769600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 981/1956] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:18 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:18.997483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:18.998965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:19.011388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4464a90-9941-42f3-91c6-fa78f5068bb1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:19.014942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e281b757-f8e7-4aca-b5ef-96758a5e01a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:19.015592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281b757-f8e7-4aca-b5ef-96758a5e01a1 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:19.015879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281b757-f8e7-4aca-b5ef-96758a5e01a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:19.016159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281b757-f8e7-4aca-b5ef-96758a5e01a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:19.075942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281b757-f8e7-4aca-b5ef-96758a5e01a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:19.077423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281b757-f8e7-4aca-b5ef-96758a5e01a1 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:19.086358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 976/1957] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:18 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:28.765373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=36,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:29.082657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=94 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:42.550115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:42.551516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:42.621185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc981a77-a8bc-474d-9944-cec0e4acff01 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:42.626474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:42.627525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:42.627925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:42.628270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] PATH_INFO: `/v3/users/e18b0b5e0e244b4e94d7f58c3c9127b1` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:42.630219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e18b0b5e0e244b4e94d7f58c3c9127b1)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:42.632278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:42.687221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e18b0b5e0e244b4e94d7f58c3c9127b1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:42.687801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e18b0b5e0e244b4e94d7f58c3c9127b1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:42.695070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e18b0b5e0e244b4e94d7f58c3c9127b1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:42.695823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e18b0b5e0e244b4e94d7f58c3c9127b1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:42.707075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e18b0b5e0e244b4e94d7f58c3c9127b1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:42.725752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 982/1958] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:42 2022] DELETE /identity/v3/users/e18b0b5e0e244b4e94d7f58c3c9127b1 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:42.730925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:42.731417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:42.731547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:42.750022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:42.751244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:42.815545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 977/1959] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:42.825687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:42.827602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:42.907565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8304b4b5-3d27-4639-8687-f2cd64fc88d8 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:42.920834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:42.921716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:42.922095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:42.922476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:42.934821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:42.936377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:42.946605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 983/1960] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:43.174074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:43.175646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:43.190213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704a864d-90cf-4ec1-9bd4-e4c431a4eeb9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:43.195160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:43.196129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:43.196447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:43.196801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] PATH_INFO: `/v3/users/65efedb96a1f4c9d84ec15607b1fff92` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:43.197951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=65efedb96a1f4c9d84ec15607b1fff92)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:43.199966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:43.252614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65efedb96a1f4c9d84ec15607b1fff92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.264495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65efedb96a1f4c9d84ec15607b1fff92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.265493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65efedb96a1f4c9d84ec15607b1fff92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.266286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65efedb96a1f4c9d84ec15607b1fff92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.271995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:43.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:43.282493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-71a5c612-69ea-4690-b641-3423c0e536ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65efedb96a1f4c9d84ec15607b1fff92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.301064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 978/1961] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:43 2022] DELETE /identity/v3/users/65efedb96a1f4c9d84ec15607b1fff92 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:43.305667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:43.306054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:43.306342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:43.309461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925323b3-73f9-46ae-8969-8fd6f600de30 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:43.327249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:43.329150 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:43.338550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:43.400006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 979/1962] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:11:43.407570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:43.409058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:43.413196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:43.414560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:43.415028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:43.415406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] PATH_INFO: `/v3/projects/75cf728448e34dc3a6c4714a0b793caf` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:43.417858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75cf728448e34dc3a6c4714a0b793caf)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:43.421004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:43.432144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:43.499487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a428cf2-6bc4-48b0-be1c-8c7abbc0afcd None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:43.503178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75cf728448e34dc3a6c4714a0b793caf', 'request_id': 'req-2c8bd159-48c4-4ce1-94ff-e0746f04f882'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.513412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75cf728448e34dc3a6c4714a0b793caf', 'request_id': 'req-2c8bd159-48c4-4ce1-94ff-e0746f04f882'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.529007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75cf728448e34dc3a6c4714a0b793caf', 'request_id': 'req-2c8bd159-48c4-4ce1-94ff-e0746f04f882'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.546257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] The token cache is being invalidate because project 75cf728448e34dc3a6c4714a0b793caf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:43.546730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.547896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.550205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 984/1963] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:43 2022] DELETE /identity/v3/projects/75cf728448e34dc3a6c4714a0b793caf => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:11:43.554870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:43.558006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:43.618293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:43.619160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:43.619457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:43.619742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:43.626589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:43.750003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8bd159-48c4-4ce1-94ff-e0746f04f882 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:43.761823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:43.763290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:43.764303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:43.765450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:43.765742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:43.775181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] PATH_INFO: `/v3/users/10da914f18ca4897aa36efb2d19e0898` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:43.778693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] RBAC: Authorizing `identity:delete_user(user_id=10da914f18ca4897aa36efb2d19e0898)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:43.782564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:43.783774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 980/1964] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:43.853011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10da914f18ca4897aa36efb2d19e0898'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.853697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10da914f18ca4897aa36efb2d19e0898'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.860551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:43.863627 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10da914f18ca4897aa36efb2d19e0898'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.865360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10da914f18ca4897aa36efb2d19e0898'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.877963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b59c517f-53aa-4994-9182-80bab164006a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10da914f18ca4897aa36efb2d19e0898'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:43.905408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 985/1965] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:43 2022] DELETE /identity/v3/users/10da914f18ca4897aa36efb2d19e0898 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:44.149837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:44.152557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.168611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83eba8ba-7c07-4e50-971f-40bd84c8fc10 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:44.239263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:44.240173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.240492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.240864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] PATH_INFO: `/v3/users/ad941a8952cb489fb18a2bebdc9319d2` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.241997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad941a8952cb489fb18a2bebdc9319d2)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:44.256053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:44.270611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:44.306171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad941a8952cb489fb18a2bebdc9319d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.315278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad941a8952cb489fb18a2bebdc9319d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.316004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad941a8952cb489fb18a2bebdc9319d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.316645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad941a8952cb489fb18a2bebdc9319d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.331127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad941a8952cb489fb18a2bebdc9319d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.352920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 981/1966] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:44 2022] DELETE /identity/v3/users/ad941a8952cb489fb18a2bebdc9319d2 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:44.357659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91cbe32-50e3-4189-93af-e4f5f0582b69 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.357950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91cbe32-50e3-4189-93af-e4f5f0582b69 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.358233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91cbe32-50e3-4189-93af-e4f5f0582b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.380849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b91cbe32-50e3-4189-93af-e4f5f0582b69 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:44.382202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b91cbe32-50e3-4189-93af-e4f5f0582b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.470682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 986/1967] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:44.524021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:44.525675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.545317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc5ee00-0930-4fb4-a3fb-c13ff2af9e80 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:44.550323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:44.551307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.551660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.552041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] PATH_INFO: `/v3/projects/c17673bc04044403a24371ae9097a572` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.553913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c17673bc04044403a24371ae9097a572)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:44.556243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:44.606835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c17673bc04044403a24371ae9097a572', 'request_id': 'req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.616176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c17673bc04044403a24371ae9097a572', 'request_id': 'req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.623883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c17673bc04044403a24371ae9097a572', 'request_id': 'req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.652674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] The token cache is being invalidate because project c17673bc04044403a24371ae9097a572 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:44.653579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.654383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:44.659627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 982/1968] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:44 2022] DELETE /identity/v3/projects/c17673bc04044403a24371ae9097a572 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:44.661481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3695b6f6-373e-4631-aa80-ef0be2da2ac4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.661955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3695b6f6-373e-4631-aa80-ef0be2da2ac4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.662176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3695b6f6-373e-4631-aa80-ef0be2da2ac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.682356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-3695b6f6-373e-4631-aa80-ef0be2da2ac4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:44.683678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3695b6f6-373e-4631-aa80-ef0be2da2ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.695204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:44.697058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.770388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 987/1969] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:44.779872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b40b649-86ac-4a09-9c08-500cdabb09f5 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.780514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b40b649-86ac-4a09-9c08-500cdabb09f5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.780989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b40b649-86ac-4a09-9c08-500cdabb09f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.783387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee97a466-f242-44fb-8f38-7d8a8f21f2e9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:44.787688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:44.788649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.789091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.789447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.791469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.812013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:44.813633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:44.816163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2b40b649-86ac-4a09-9c08-500cdabb09f5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:44.818107 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2b40b649-86ac-4a09-9c08-500cdabb09f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.825327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 983/1970] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:11:44.831010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:44.832534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 988/1971] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:44.835347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:44.836966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b40b649-86ac-4a09-9c08-500cdabb09f5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:44.847493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b40b649-86ac-4a09-9c08-500cdabb09f5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:44.859389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7043c105-fd22-420f-905f-3945a7c5554e service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:44.860671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:44.861385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.861666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.861991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.862782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:11:44.863096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:44.865302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:44.876636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:44.878866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.879937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.880471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] PATH_INFO: `/v3/users/980825b12301415197171fcabcbac199` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.881161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 989/1972] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:11:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:11:44.882401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] RBAC: Authorizing `identity:delete_user(user_id=980825b12301415197171fcabcbac199)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:44.884675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:44.885776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:44.899296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7b71c8-dc73-4614-90f8-d80c2d830f9f admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:44.912652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:44.913631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:44.914121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:44.914604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:44.915874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:44.918694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:44.935714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:44.962840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 990/1973] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:44 2022] POST /identity/v3/projects => generated 385 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:11:44.964894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:44.994026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '980825b12301415197171fcabcbac199'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.003277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '980825b12301415197171fcabcbac199'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.004512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '980825b12301415197171fcabcbac199'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.005328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '980825b12301415197171fcabcbac199'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.007150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:45.030175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dbe4352a-8813-44bd-8d6e-2c91e822c424 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '980825b12301415197171fcabcbac199'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.041290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe342899-6d49-4187-b671-d9a02491be5b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.050198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.051044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.051435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.052729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.053588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.054618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 984/1974] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:44 2022] DELETE /identity/v3/users/980825b12301415197171fcabcbac199 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:11:45.058631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.061127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.061434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.061712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.089386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:45.091282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:45.106641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 991/1975] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:44 2022] POST /identity/v3/users => generated 293 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:45.109885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.114324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:45.132313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4cb8cb-6fd6-44ff-aab6-f03f0755a8f4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.205470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.206432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.206818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.207282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.208500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 985/1976] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:45.208631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.209747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.218510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.220239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:45.225556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 992/1977] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:45.230659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.244952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51c3687-07fb-4286-818a-c6d63db11147 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.251351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.252137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.252524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.252871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.253678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.255416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.268764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 993/1978] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:45.272909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.283681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d19ae4-64ad-4ad1-bd8c-2165f370e4b2 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.290662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34922cc-bd54-482e-be7c-3d35d64c0a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.290662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34922cc-bd54-482e-be7c-3d35d64c0a60 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.290662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34922cc-bd54-482e-be7c-3d35d64c0a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.290662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34922cc-bd54-482e-be7c-3d35d64c0a60 admin admin] PATH_INFO: `/v3/projects/865c43972cab49f8b1295f8d3c2dae9c/users/a9cb4268e7eb40688f331a32dd8a11db/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.297883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf3beda-810e-4feb-a9d8-d594ab378fd1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.302449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.303288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.303904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.304328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.309546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34922cc-bd54-482e-be7c-3d35d64c0a60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=865c43972cab49f8b1295f8d3c2dae9c, user_id=a9cb4268e7eb40688f331a32dd8a11db, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.312359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34922cc-bd54-482e-be7c-3d35d64c0a60 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.315805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.317293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.329339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 994/1979] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:45 2022] PUT /identity/v3/projects/865c43972cab49f8b1295f8d3c2dae9c/users/a9cb4268e7eb40688f331a32dd8a11db/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:11:45.331357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 986/1980] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:11:45.345599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.346045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.346424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.384590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] MFA Rules not processed for user `a9cb4268e7eb40688f331a32dd8a11db`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:45.386698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:45.492072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 995/1981] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:45 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:45.494296 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.508874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b595e72-ef5f-49f0-86b8-b67d28fe1d48 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.535753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.538007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:45.571089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:45.604484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:45.636649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.637672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.637978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.638344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.639017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.641319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.655614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84dadfe-0651-4297-9bb3-a6f174a26239 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.662100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.663164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.663695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.663695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] PATH_INFO: `/v3/projects/9de50fd47c1f4c639b5cf0c1ebb49b96` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.665080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9de50fd47c1f4c639b5cf0c1ebb49b96)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.667035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.677002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 987/1982] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:45 2022] POST /identity/v3/projects => generated 385 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:45.680627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.753792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9de50fd47c1f4c639b5cf0c1ebb49b96', 'request_id': 'req-a012e532-2433-4f7b-8f15-5e130ddf6954'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.762365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9de50fd47c1f4c639b5cf0c1ebb49b96', 'request_id': 'req-a012e532-2433-4f7b-8f15-5e130ddf6954'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.769458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7957746b-e89f-45f5-9445-e510c140dc7b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.785264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9de50fd47c1f4c639b5cf0c1ebb49b96', 'request_id': 'req-a012e532-2433-4f7b-8f15-5e130ddf6954'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.799360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] The token cache is being invalidate because project 9de50fd47c1f4c639b5cf0c1ebb49b96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:45.799714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.803554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.805016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 996/1983] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:45 2022] DELETE /identity/v3/projects/9de50fd47c1f4c639b5cf0c1ebb49b96 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:45.809346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.811056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:45.835975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.836693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.836978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.837255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.837951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.840546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.868293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 988/1984] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:45 2022] POST /identity/v3/users => generated 299 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:11:45.869239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:45.871031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:45.883377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57e4454-5568-4044-a1b8-0d5ef7dbac25 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.889054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.889934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.890215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.890496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] PATH_INFO: `/v3/users/531bce66ddc8487cb390029af2ba3587` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.891800 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] RBAC: Authorizing `identity:delete_user(user_id=531bce66ddc8487cb390029af2ba3587)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.893778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.894187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a012e532-2433-4f7b-8f15-5e130ddf6954 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:45.900865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:45.902209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:45.902497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:45.902865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] PATH_INFO: `/v3/projects/e21303cf10464bdf9e2936db97f86792` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:45.904114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e21303cf10464bdf9e2936db97f86792)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:45.906172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:45.965372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '531bce66ddc8487cb390029af2ba3587'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.966280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:45.972237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e21303cf10464bdf9e2936db97f86792', 'request_id': 'req-0221a171-3902-40e2-a890-a986d40f5af7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.975530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '531bce66ddc8487cb390029af2ba3587'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.976383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '531bce66ddc8487cb390029af2ba3587'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.976983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '531bce66ddc8487cb390029af2ba3587'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.978258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:45.980468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e21303cf10464bdf9e2936db97f86792', 'request_id': 'req-0221a171-3902-40e2-a890-a986d40f5af7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.990343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e21303cf10464bdf9e2936db97f86792', 'request_id': 'req-0221a171-3902-40e2-a890-a986d40f5af7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:45.995332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '531bce66ddc8487cb390029af2ba3587'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.006864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] The token cache is being invalidate because project e21303cf10464bdf9e2936db97f86792 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:46.007081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.007858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.009601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 997/1985] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:45 2022] DELETE /identity/v3/projects/e21303cf10464bdf9e2936db97f86792 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:46.010541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.013461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.019929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 989/1986] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:45 2022] DELETE /identity/v3/users/531bce66ddc8487cb390029af2ba3587 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:46.019929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.023012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.144092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0221a171-3902-40e2-a890-a986d40f5af7 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.149611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d790e1a1-3c69-4c2c-a424-32ce9ea0c15e admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.155280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.156258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.158238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.158977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.159171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.159398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.159685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] PATH_INFO: `/v3/users/d7a6bf0045f94f64b4ef8ad9e0164e93` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.160066 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.161278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7a6bf0045f94f64b4ef8ad9e0164e93)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.161845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.163720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:46.164535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:46.194090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 998/1987] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:46 2022] GET /identity/v3/roles => generated 1420 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:46.196610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.210801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b21ce-fe33-4908-8c7a-d6865c75df8c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.227563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7a6bf0045f94f64b4ef8ad9e0164e93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.239459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7a6bf0045f94f64b4ef8ad9e0164e93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.240261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7a6bf0045f94f64b4ef8ad9e0164e93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.241799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7a6bf0045f94f64b4ef8ad9e0164e93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.242618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.246788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.249606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.252620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] PATH_INFO: `/v3/projects/446c177fc4c9405faefd4ebc7baa0dd4/users/4defad3ef01e408e805c48f0de9c6f82/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.260718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7a6bf0045f94f64b4ef8ad9e0164e93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.280287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=446c177fc4c9405faefd4ebc7baa0dd4, user_id=4defad3ef01e408e805c48f0de9c6f82, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.281732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 990/1988] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:46 2022] DELETE /identity/v3/users/d7a6bf0045f94f64b4ef8ad9e0164e93 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:46.284832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:46.303346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 999/1989] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:46 2022] PUT /identity/v3/projects/446c177fc4c9405faefd4ebc7baa0dd4/users/4defad3ef01e408e805c48f0de9c6f82/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:46.307163 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.308786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.372089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:46.392219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f893c23-71aa-4a8a-a5ed-5cca85fd3643 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.468884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.470803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.509552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.512748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.513271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.513788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.514897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.518042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:46.548887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a77a119-1a8f-4584-9078-7f392b8cefb0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.558366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.559569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.559960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.560339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] PATH_INFO: `/v3/projects/c783e271fb47447d9a8e63c8c3e827bf` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.563147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c783e271fb47447d9a8e63c8c3e827bf)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.564646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:46.569594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 991/1990] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:46 2022] GET /identity/v3/roles => generated 1420 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:46.570904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.587869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fdfff4-c5d6-4078-ac2d-d0a788bed3c0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.595438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.596602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.597219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.597661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] PATH_INFO: `/v3/projects/446c177fc4c9405faefd4ebc7baa0dd4/users/4defad3ef01e408e805c48f0de9c6f82/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.601147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=446c177fc4c9405faefd4ebc7baa0dd4, user_id=4defad3ef01e408e805c48f0de9c6f82, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.604987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:46.619830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 992/1991] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:46 2022] PUT /identity/v3/projects/446c177fc4c9405faefd4ebc7baa0dd4/users/4defad3ef01e408e805c48f0de9c6f82/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:46.621323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.626284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.629914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:46.650756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c783e271fb47447d9a8e63c8c3e827bf', 'request_id': 'req-9e0428f5-610e-4a8e-954f-baabf8e17c4f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.676051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c783e271fb47447d9a8e63c8c3e827bf', 'request_id': 'req-9e0428f5-610e-4a8e-954f-baabf8e17c4f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.691589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c783e271fb47447d9a8e63c8c3e827bf', 'request_id': 'req-9e0428f5-610e-4a8e-954f-baabf8e17c4f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.710547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] The token cache is being invalidate because project c783e271fb47447d9a8e63c8c3e827bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:46.710829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.711666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9e0428f5-610e-4a8e-954f-baabf8e17c4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.713974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1000/1992] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:46 2022] DELETE /identity/v3/projects/c783e271fb47447d9a8e63c8c3e827bf => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:46.716049 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.716452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.716763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.728181 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3cbeb2-35a4-443a-88f5-c8b962c946f5 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.754370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] MFA Rules not processed for user `4defad3ef01e408e805c48f0de9c6f82`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:46.756799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.809959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.810875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.811234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.811681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] PATH_INFO: `/v3/projects/6c4a5c789e384a7e88a70b57bdc3a619` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.813257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c4a5c789e384a7e88a70b57bdc3a619)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.815659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:46.856516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1001/1993] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:46 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:46.856516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.858329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:46.861091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.871511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c4a5c789e384a7e88a70b57bdc3a619', 'request_id': 'req-cd649aba-cffd-428f-8c2d-1a75474543cc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.877901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c4a5c789e384a7e88a70b57bdc3a619', 'request_id': 'req-cd649aba-cffd-428f-8c2d-1a75474543cc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.879306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b90bae9-ccea-4b0c-a311-84f4b00b88e4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:46.888336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c4a5c789e384a7e88a70b57bdc3a619', 'request_id': 'req-cd649aba-cffd-428f-8c2d-1a75474543cc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.905076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] The token cache is being invalidate because project 6c4a5c789e384a7e88a70b57bdc3a619 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:46.905904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.907115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:46.909463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 993/1994] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:46 2022] DELETE /identity/v3/projects/6c4a5c789e384a7e88a70b57bdc3a619 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:46.910409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:46.912326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:46.979442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:46.980517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:46.980874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:46.981300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] PATH_INFO: `/v3/users/8dc8994198ed429c85b7e32b60a5063a` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:46.982859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8dc8994198ed429c85b7e32b60a5063a)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:46.985278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.001415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd649aba-cffd-428f-8c2d-1a75474543cc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.005595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.006253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.006538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.006843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.009078 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.067692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8dc8994198ed429c85b7e32b60a5063a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.068372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8dc8994198ed429c85b7e32b60a5063a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.090613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8dc8994198ed429c85b7e32b60a5063a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.091341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8dc8994198ed429c85b7e32b60a5063a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.095928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:47.099960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f23ba7a7-c6cf-4514-8b10-776e2ea5c751 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8dc8994198ed429c85b7e32b60a5063a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.130129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1002/1995] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:46 2022] DELETE /identity/v3/users/8dc8994198ed429c85b7e32b60a5063a => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:47.131746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.131746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.131963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.149575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.151497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.153358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:47.154576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.169458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 994/1996] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:46 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:47.170532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.172275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.194227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14771ae-4e4b-4c8b-a324-0536b20f1f07 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.236478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1003/1997] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:47.237166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.238453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.252485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed47ba23-90a8-4a2d-a509-6d4d7907cda3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.258058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.258941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.259224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.259502 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] PATH_INFO: `/v3/users/a9cb4268e7eb40688f331a32dd8a11db` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.260761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9cb4268e7eb40688f331a32dd8a11db)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.262742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.266976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.268058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.268574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.269058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] PATH_INFO: `/v3/projects/1b907107c8f8461d92b6a2daf658c9ee` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.270632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b907107c8f8461d92b6a2daf658c9ee)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.272817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.286028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:47.326288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9cb4268e7eb40688f331a32dd8a11db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.327241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9cb4268e7eb40688f331a32dd8a11db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.335854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9cb4268e7eb40688f331a32dd8a11db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.336033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b907107c8f8461d92b6a2daf658c9ee', 'request_id': 'req-1910c065-3854-4e1c-97ac-76a8db66a6bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.336631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9cb4268e7eb40688f331a32dd8a11db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.343894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b907107c8f8461d92b6a2daf658c9ee', 'request_id': 'req-1910c065-3854-4e1c-97ac-76a8db66a6bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.345667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-9072d365-7e16-4041-ae78-18184e3b35b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9cb4268e7eb40688f331a32dd8a11db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.350561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b907107c8f8461d92b6a2daf658c9ee', 'request_id': 'req-1910c065-3854-4e1c-97ac-76a8db66a6bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.366425 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] The token cache is being invalidate because project 1b907107c8f8461d92b6a2daf658c9ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:47.366939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.370996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1910c065-3854-4e1c-97ac-76a8db66a6bd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.371118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1004/1998] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:47 2022] DELETE /identity/v3/users/a9cb4268e7eb40688f331a32dd8a11db => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:47.371118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50095dfa-5468-450e-af95-7c6f8ed6118f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.371118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50095dfa-5468-450e-af95-7c6f8ed6118f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.371118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50095dfa-5468-450e-af95-7c6f8ed6118f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.372588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 995/1999] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:47 2022] DELETE /identity/v3/projects/1b907107c8f8461d92b6a2daf658c9ee => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:47.377169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19032c71-d05d-41d0-b147-bcfbe129eea2 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.377887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19032c71-d05d-41d0-b147-bcfbe129eea2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.378385 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19032c71-d05d-41d0-b147-bcfbe129eea2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.394469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-50095dfa-5468-450e-af95-7c6f8ed6118f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:47.395698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-50095dfa-5468-450e-af95-7c6f8ed6118f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.398971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-19032c71-d05d-41d0-b147-bcfbe129eea2 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:47.399720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-19032c71-d05d-41d0-b147-bcfbe129eea2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.473797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:47.495229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1005/2000] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:47.495446 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 996/2001] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:47.495532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8976af93-ec62-4061-8d6b-61dffa08d13d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.495532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8976af93-ec62-4061-8d6b-61dffa08d13d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.495532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8976af93-ec62-4061-8d6b-61dffa08d13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.497817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19032c71-d05d-41d0-b147-bcfbe129eea2 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.507968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19032c71-d05d-41d0-b147-bcfbe129eea2 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.513327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.514030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.514374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.514595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.515280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8976af93-ec62-4061-8d6b-61dffa08d13d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:47.515394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:11:47.515932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.516515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8976af93-ec62-4061-8d6b-61dffa08d13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.517284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.530764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 997/2002] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:11:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:11:47.532080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.535787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.543183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1006/2003] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:47.544079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8976af93-ec62-4061-8d6b-61dffa08d13d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.556185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8976af93-ec62-4061-8d6b-61dffa08d13d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.567691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63657e37-a0b0-4a93-88fc-34c6fc8551e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.568372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63657e37-a0b0-4a93-88fc-34c6fc8551e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.568690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63657e37-a0b0-4a93-88fc-34c6fc8551e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.568971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63657e37-a0b0-4a93-88fc-34c6fc8551e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.569804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63657e37-a0b0-4a93-88fc-34c6fc8551e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.572294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63657e37-a0b0-4a93-88fc-34c6fc8551e1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.608216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1007/2004] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/projects => generated 365 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:47.610150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1e7cc5-e439-40ef-a3fc-064fd389ce87 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.611410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1e7cc5-e439-40ef-a3fc-064fd389ce87 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.611806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1e7cc5-e439-40ef-a3fc-064fd389ce87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.620643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1dd84d-5659-4aa2-a28d-c6c146a72901 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.671909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ac1e7cc5-e439-40ef-a3fc-064fd389ce87 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:47.673681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ac1e7cc5-e439-40ef-a3fc-064fd389ce87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.710434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:47.724617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.725597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.726312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.726631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.763073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1008/2005] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:47.764262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1e7cc5-e439-40ef-a3fc-064fd389ce87 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.776169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1e7cc5-e439-40ef-a3fc-064fd389ce87 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.781856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.782617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.783084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.783421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.784172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.786222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.805362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.806808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.812818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1009/2006] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/users => generated 283 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:47.813799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.821727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 998/2007] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:47.822441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.824417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:47.825094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d049cf46-b1a9-40c0-9036-6f6db6c7d64c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.830110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.830781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.831069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.831344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.832085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:11:47.832763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.835933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.837684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa41c259-f09d-4f9f-99fe-87a4cc034eae service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.843321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.844214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.844533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.844818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] PATH_INFO: `/v3/projects/e0f93936e3444cab851397ace68a6341` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.846138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0f93936e3444cab851397ace68a6341)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.848359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.865421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1010/2008] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:11:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:11:47.866245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.883721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daf781d-3652-4663-a892-8bea07b00cad admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.889816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:47.890915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:47.890915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:47.891133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:47.891842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:47.894300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:47.909154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1011/2009] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:47.909896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.918613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0f93936e3444cab851397ace68a6341', 'request_id': 'req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.926663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0f93936e3444cab851397ace68a6341', 'request_id': 'req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.936786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9802e85-7606-4a56-8367-2d49acbfc1af admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:47.940108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0f93936e3444cab851397ace68a6341', 'request_id': 'req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.958724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] The token cache is being invalidate because project e0f93936e3444cab851397ace68a6341 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:47.959842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.961323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:47.969094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 999/2010] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:47 2022] DELETE /identity/v3/projects/e0f93936e3444cab851397ace68a6341 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:47.969997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:47.971378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.026563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc489c47-5c99-4e1e-8b70-afd159963e46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.027287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc489c47-5c99-4e1e-8b70-afd159963e46 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.031568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc489c47-5c99-4e1e-8b70-afd159963e46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.032054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc489c47-5c99-4e1e-8b70-afd159963e46 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.032979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc489c47-5c99-4e1e-8b70-afd159963e46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.035573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc489c47-5c99-4e1e-8b70-afd159963e46 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.077471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1012/2011] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:47 2022] POST /identity/v3/projects => generated 375 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:48.079099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.079583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.080029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.103278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:48.112546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:48.115619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.120159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e13fc2f-6b9a-4f19-b2b7-0dafc380bd0e admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.150691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:48.215608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1013/2012] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:48.216551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.217250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.217533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.218068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.219326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.224217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.228744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.237341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5e1610-00fd-46c0-a7e0-2e2ea3febf4f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.242161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ba4f15-9f8f-4869-a010-61ebf7fb4880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.242890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba4f15-9f8f-4869-a010-61ebf7fb4880 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.243174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba4f15-9f8f-4869-a010-61ebf7fb4880 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.243453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ba4f15-9f8f-4869-a010-61ebf7fb4880 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.244138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba4f15-9f8f-4869-a010-61ebf7fb4880 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.245899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1000/2013] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:47 2022] GET /identity/v3/roles => generated 1420 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:48.246256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ba4f15-9f8f-4869-a010-61ebf7fb4880 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.246865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.248850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.261684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88a6d05-0297-4c11-a3e5-398a1962f16f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.266656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.267527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.267814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.268170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] PATH_INFO: `/v3/projects/865c43972cab49f8b1295f8d3c2dae9c` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.269466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=865c43972cab49f8b1295f8d3c2dae9c)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.273944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1014/2014] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:48 2022] POST /identity/v3/users => generated 288 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:48.274093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.274187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9785e-0e55-41e3-bcc1-08d32c8ee3f8 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.274187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9785e-0e55-41e3-bcc1-08d32c8ee3f8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.274187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd9785e-0e55-41e3-bcc1-08d32c8ee3f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.294769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2dd9785e-0e55-41e3-bcc1-08d32c8ee3f8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:48.295999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2dd9785e-0e55-41e3-bcc1-08d32c8ee3f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.305851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1015/2015] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:48.306698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9785e-0e55-41e3-bcc1-08d32c8ee3f8 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.318245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd9785e-0e55-41e3-bcc1-08d32c8ee3f8 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.326606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '865c43972cab49f8b1295f8d3c2dae9c', 'request_id': 'req-4dce1202-4f09-4a73-b307-fafa471ebaa7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.332350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '865c43972cab49f8b1295f8d3c2dae9c', 'request_id': 'req-4dce1202-4f09-4a73-b307-fafa471ebaa7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.338828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '865c43972cab49f8b1295f8d3c2dae9c', 'request_id': 'req-4dce1202-4f09-4a73-b307-fafa471ebaa7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.352726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] The token cache is being invalidate because project 865c43972cab49f8b1295f8d3c2dae9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:48.353014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.353667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.359017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1001/2016] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:48 2022] DELETE /identity/v3/projects/865c43972cab49f8b1295f8d3c2dae9c => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:48.359017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.360429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.387473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.388282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.388753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.388864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] PATH_INFO: `/v3/projects/df29dee9940b489b85acbae532d9ad62/users/1e09f089d0a54814bf92eac518ad42f0/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.406262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df29dee9940b489b85acbae532d9ad62, user_id=1e09f089d0a54814bf92eac518ad42f0, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.408722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.422722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1016/2017] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:48 2022] PUT /identity/v3/projects/df29dee9940b489b85acbae532d9ad62/users/1e09f089d0a54814bf92eac518ad42f0/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:48.423751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.425357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.448455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dce1202-4f09-4a73-b307-fafa471ebaa7 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.509990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:48.520243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3402e5f4-c3b5-42ac-9041-9e6164e16df6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.527241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.528129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.528443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.528873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.529628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:11:48.529945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.531264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4279d927-fb17-40b0-9162-c1cb6e0db2d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.531626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.531876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279d927-fb17-40b0-9162-c1cb6e0db2d0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.532165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279d927-fb17-40b0-9162-c1cb6e0db2d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.532459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4279d927-fb17-40b0-9162-c1cb6e0db2d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.533203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4279d927-fb17-40b0-9162-c1cb6e0db2d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.536375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4279d927-fb17-40b0-9162-c1cb6e0db2d0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.548549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:48.548549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1017/2018] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:11:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:11:48.548549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.550704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.555394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1002/2019] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:48.556897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.557193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.557472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.569113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33e48d5-d161-4d86-8455-d617fac23d70 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.577244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.578623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.579103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.579716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] PATH_INFO: `/v3/users/4defad3ef01e408e805c48f0de9c6f82` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.581235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] RBAC: Authorizing `identity:delete_user(user_id=4defad3ef01e408e805c48f0de9c6f82)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.583999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.603322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] MFA Rules not processed for user `1e09f089d0a54814bf92eac518ad42f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:48.604689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.645954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4defad3ef01e408e805c48f0de9c6f82'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.646707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4defad3ef01e408e805c48f0de9c6f82'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.654854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4defad3ef01e408e805c48f0de9c6f82'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.655537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4defad3ef01e408e805c48f0de9c6f82'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.664237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4defad3ef01e408e805c48f0de9c6f82'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:48.690516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1018/2020] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:11:48 2022] DELETE /identity/v3/users/4defad3ef01e408e805c48f0de9c6f82 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:48.691713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.693259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.709400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1003/2021] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:48 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:48.710801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.712338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.732230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37c8ae1-e3db-4186-b637-fbba6942a22a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.792454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b28c207-dd25-42c8-aa2b-6496c573020b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.797900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.798630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.798915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.799346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.799981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.802494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.842176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.842898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.843184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.843455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.844195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.846457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:48.849590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1019/2022] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:48 2022] POST /identity/v3/projects => generated 379 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:48.850477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.852724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:48.868611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1004/2023] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:48.868611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:48.893353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca6f6a4-0a07-4dff-a4c5-76005c79bbb3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.943566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:48.952449 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78336476-b89b-4dbf-a831-509572ee2ce4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:48.962977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:48.963660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:48.963949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:48.964830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:48.965541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:48.967485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.002447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.003034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1020/2024] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:48 2022] POST /identity/v3/projects => generated 363 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:49.003118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.003190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.003261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.003261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.003261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.004613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.032553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1005/2025] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:48 2022] POST /identity/v3/users => generated 290 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:49.034530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.052758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:49.065289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e37fe2-37e3-4506-be17-8c88e219b8fe admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.119085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f95d1-063e-47d2-880f-b15b90c9230e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.132526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.133844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.134160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.134450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] PATH_INFO: `/v3/projects/39d74b57b467479f8fb7e0f44d937163/users/302c737b117842fa9faed06f0f8331d0/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.160186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.161184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.161397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.161687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.162366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.163051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39d74b57b467479f8fb7e0f44d937163, user_id=302c737b117842fa9faed06f0f8331d0, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.164318 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.165830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.180933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1021/2026] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:48 2022] PUT /identity/v3/projects/39d74b57b467479f8fb7e0f44d937163/users/302c737b117842fa9faed06f0f8331d0/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:49.185386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.189151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1006/2027] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:49 2022] POST /identity/v3/users => generated 288 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:49.189151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.191347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.203783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d284651-81ac-4e49-8a27-0b80d4189a10 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.253367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4538b5-1644-4215-accb-9cb8c8a57110 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.258139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.258801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.259091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.259368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.259740 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.260092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.260632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.260926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.261792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] PATH_INFO: `/v3/projects/446c177fc4c9405faefd4ebc7baa0dd4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.262446 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.262997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] RBAC: Authorizing `identity:delete_project(project_id=446c177fc4c9405faefd4ebc7baa0dd4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.265029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.272537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1022/2028] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:49 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:49.273162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.282621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1f0210-886e-489d-9a42-cf2ecba56ac4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.287174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121003a4-fe2a-407f-8871-4b334def19b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.287810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121003a4-fe2a-407f-8871-4b334def19b4 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.288092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121003a4-fe2a-407f-8871-4b334def19b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.288373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121003a4-fe2a-407f-8871-4b334def19b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.289112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121003a4-fe2a-407f-8871-4b334def19b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.290736 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121003a4-fe2a-407f-8871-4b334def19b4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.300760 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1023/2029] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:49 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:49.302089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.302375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.302804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.313863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '446c177fc4c9405faefd4ebc7baa0dd4', 'request_id': 'req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:49.319802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '446c177fc4c9405faefd4ebc7baa0dd4', 'request_id': 'req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:49.327307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '446c177fc4c9405faefd4ebc7baa0dd4', 'request_id': 'req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:49.329973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] MFA Rules not processed for user `302c737b117842fa9faed06f0f8331d0`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:49.331174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.340046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] The token cache is being invalidate because project 446c177fc4c9405faefd4ebc7baa0dd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:11:49.340337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:49.341322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:11:49.343166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1007/2030] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:11:49 2022] DELETE /identity/v3/projects/446c177fc4c9405faefd4ebc7baa0dd4 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:49.344041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.345335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.372834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:49.373643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:49.401112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1024/2031] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:49 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:49.402072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.403417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.416670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9685e20-7b32-4aa9-9ec3-de9e65fb0765 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.421869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc7f32-1f3b-4a90-8aeb-3390e5356b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.422548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7f32-1f3b-4a90-8aeb-3390e5356b1c admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.422839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7f32-1f3b-4a90-8aeb-3390e5356b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.423120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7f32-1f3b-4a90-8aeb-3390e5356b1c admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.423852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fc7f32-1f3b-4a90-8aeb-3390e5356b1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.425523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fc7f32-1f3b-4a90-8aeb-3390e5356b1c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.440954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1008/2032] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:49 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:49.440954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.440954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.440954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.457453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:49.458881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.469803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1009/2033] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:49.470293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf46b137-d6f4-462d-813f-786f85cf424a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.470616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.471933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.480571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.481546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.481832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.482109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] PATH_INFO: `/v3/projects/18efaeb5acbd4def8945616c030aaa9c/users/42492edb372d4887bc6a70bb3df6b0ed/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.498732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18efaeb5acbd4def8945616c030aaa9c, user_id=42492edb372d4887bc6a70bb3df6b0ed, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.501459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.520495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1025/2034] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:49 2022] PUT /identity/v3/projects/18efaeb5acbd4def8945616c030aaa9c/users/42492edb372d4887bc6a70bb3df6b0ed/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:49.521596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.538095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b90aa4-0d15-4cf3-a2c6-119f66fe308b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.592651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d9118e-b44f-482a-a224-4deaea08dca6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.593874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.594568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.594859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.595142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.598174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66d362c-1b4c-4cf0-80fd-4531003c5351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.599603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66d362c-1b4c-4cf0-80fd-4531003c5351 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.601217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66d362c-1b4c-4cf0-80fd-4531003c5351 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.602032 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66d362c-1b4c-4cf0-80fd-4531003c5351 admin admin] PATH_INFO: `/v3/projects/37d728b7da6b4b52aec0cdb5cb45bb79/users/fadb4399d54543a6bfea88964a1bda0f/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.623897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66d362c-1b4c-4cf0-80fd-4531003c5351 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37d728b7da6b4b52aec0cdb5cb45bb79, user_id=fadb4399d54543a6bfea88964a1bda0f, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.627835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66d362c-1b4c-4cf0-80fd-4531003c5351 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.643470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1026/2035] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:49 2022] PUT /identity/v3/projects/37d728b7da6b4b52aec0cdb5cb45bb79/users/fadb4399d54543a6bfea88964a1bda0f/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:49.644849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05b7cf-4537-4936-9191-f9885208fe65 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.645138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05b7cf-4537-4936-9191-f9885208fe65 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.645417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f05b7cf-4537-4936-9191-f9885208fe65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.665780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.667966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.669468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-3f05b7cf-4537-4936-9191-f9885208fe65 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:49.670691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3f05b7cf-4537-4936-9191-f9885208fe65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.678721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1010/2036] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:49 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:49.681461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.694255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d0da10-2672-49ae-a3b0-106c995b636c service glance] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.718789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:49.722341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:49.751884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1027/2037] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:49.756795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7131be10-f8e4-4c70-ab18-818b66548bc4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.757153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7131be10-f8e4-4c70-ab18-818b66548bc4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.757441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7131be10-f8e4-4c70-ab18-818b66548bc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.778100 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.779532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.779818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.780203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.781097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.782708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.798110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1011/2038] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:49 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:49.799166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.810590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f60809-c0a3-4794-921b-f1e8927ee9a9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.815842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.816557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.816844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.817121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.817850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:11:49.818159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.819906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.822154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7131be10-f8e4-4c70-ab18-818b66548bc4 None None] MFA Rules not processed for user `fadb4399d54543a6bfea88964a1bda0f`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:49.823900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7131be10-f8e4-4c70-ab18-818b66548bc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:49.835066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1012/2039] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:11:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:11:49.835960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.853510 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7401c71e-ccde-4cb4-9eba-c04588e7f5dd admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.861769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.862930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.863338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.864035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] PATH_INFO: `/v3/projects/18efaeb5acbd4def8945616c030aaa9c/users/42492edb372d4887bc6a70bb3df6b0ed/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.866610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18efaeb5acbd4def8945616c030aaa9c, user_id=42492edb372d4887bc6a70bb3df6b0ed, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.869420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:49.881776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1013/2040] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:49 2022] PUT /identity/v3/projects/18efaeb5acbd4def8945616c030aaa9c/users/42492edb372d4887bc6a70bb3df6b0ed/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:49.890912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:49.953527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1028/2041] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:49 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:49.953527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006488e-f2fb-458d-866c-9862927274d0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.953527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006488e-f2fb-458d-866c-9862927274d0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.953527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c006488e-f2fb-458d-866c-9862927274d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.985260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e01107-f40a-49dd-a613-15cba1abcec0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:49.994023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:49.995610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:49.995968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:49.996369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:49.997968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:49.998534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c006488e-f2fb-458d-866c-9862927274d0 None None] MFA Rules not processed for user `42492edb372d4887bc6a70bb3df6b0ed`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:49.999743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c006488e-f2fb-458d-866c-9862927274d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:50.000189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.048656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1014/2042] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:49 2022] POST /identity/v3/projects => generated 393 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:50.048656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.098188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1029/2043] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:49 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:50.099826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006488e-f2fb-458d-866c-9862927274d0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.130653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c006488e-f2fb-458d-866c-9862927274d0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.150460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.151150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.151439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.152303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.161970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.161970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.171528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb78824-aee6-4bdd-866e-12747bf7d9f8 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.183869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.185689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.185855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.186127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.187149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.189867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.191411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1030/2044] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/users => generated 297 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:50.192359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.195564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:50.232814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1015/2045] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/projects => generated 379 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:50.234270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.240609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:50.279764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:50.294563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be990d1b-642f-46ab-b1af-3bc3dc9beb63 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.349939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b005958-c696-474b-8417-2b350921b81c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.356900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.357822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.358257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.359628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.361922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.372777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.373493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.373784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.374064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.378705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1016/2046] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:50.378705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.386643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adbe20c-99d0-45ef-8ed1-63a08b76684b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.391863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.392749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.393168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.393447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.394138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.396067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.424452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1017/2047] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/users => generated 290 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:50.425299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.436226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def60c0a-279d-47fd-9ced-83e7f46d483f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.443278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.444157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.444473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.444705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.444815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.445481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.446146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.447084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.456453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1031/2048] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:50 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:50.457251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.460549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1018/2049] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:50.462753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.468383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854fbb7c-cc13-463d-a378-aaf8b84f509c service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.473359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.474005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.474299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.474863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.474971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2116f02c-af25-4e43-a7d3-31bf3434fd67 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.475292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.476910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.480507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.481466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.481754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.482064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] PATH_INFO: `/v3/projects/cc5249e571e8469d9b1bd932cc02267b/users/8be0abedb7b64f239ee7c9eb89263853/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.490105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1032/2050] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:50.495125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.504321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc5249e571e8469d9b1bd932cc02267b, user_id=8be0abedb7b64f239ee7c9eb89263853, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.506296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b226a-3022-4f3d-9491-b98c678811a4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.507191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.513967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c68a69-f8ef-4d43-8641-38fc4717a4a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.514669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c68a69-f8ef-4d43-8641-38fc4717a4a1 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.514955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c68a69-f8ef-4d43-8641-38fc4717a4a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.515410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c68a69-f8ef-4d43-8641-38fc4717a4a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.516116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c68a69-f8ef-4d43-8641-38fc4717a4a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.517759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c68a69-f8ef-4d43-8641-38fc4717a4a1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.530824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1019/2051] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:50 2022] PUT /identity/v3/projects/cc5249e571e8469d9b1bd932cc02267b/users/8be0abedb7b64f239ee7c9eb89263853/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:11:50.531052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1033/2052] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:50.535517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.538337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7fb732-497b-4ba1-9311-4f1b3aac24ac None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.538659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7fb732-497b-4ba1-9311-4f1b3aac24ac None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.538963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7fb732-497b-4ba1-9311-4f1b3aac24ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.566486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5b7fb732-497b-4ba1-9311-4f1b3aac24ac None None] MFA Rules not processed for user `8be0abedb7b64f239ee7c9eb89263853`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:50.567733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5b7fb732-497b-4ba1-9311-4f1b3aac24ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:50.590982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106a71fd-79f1-49d5-b069-000331aab1d4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.594606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.595254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.595538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.595819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.597625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:50.635776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1034/2053] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:50.640767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7fb732-497b-4ba1-9311-4f1b3aac24ac None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.664274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:50.666465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.668714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.673937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:50.680138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1020/2054] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:50 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:11:50.682031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.718555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7fb732-497b-4ba1-9311-4f1b3aac24ac None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.724166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de53902-54d0-4262-ba59-d22298a9690f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.725183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de53902-54d0-4262-ba59-d22298a9690f admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.725459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de53902-54d0-4262-ba59-d22298a9690f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.725740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de53902-54d0-4262-ba59-d22298a9690f admin admin] PATH_INFO: `/v3/projects/c55f817522494ce18875a76bb7b8d505/users/d86a609b6c0f49c882c97f30cc479988/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.744149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de53902-54d0-4262-ba59-d22298a9690f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c55f817522494ce18875a76bb7b8d505, user_id=d86a609b6c0f49c882c97f30cc479988, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.746941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de53902-54d0-4262-ba59-d22298a9690f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.764819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1035/2055] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:50 2022] PUT /identity/v3/projects/c55f817522494ce18875a76bb7b8d505/users/d86a609b6c0f49c882c97f30cc479988/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:50.771509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258c36cf-5d07-4224-aa3c-9a234c90afc7 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.776347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578e2729-5f2b-427a-8087-ee751a0105ca None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.777450 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578e2729-5f2b-427a-8087-ee751a0105ca None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.778148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578e2729-5f2b-427a-8087-ee751a0105ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.808427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-578e2729-5f2b-427a-8087-ee751a0105ca None None] MFA Rules not processed for user `d86a609b6c0f49c882c97f30cc479988`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:50.809643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-578e2729-5f2b-427a-8087-ee751a0105ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:50.837324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.838118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.838285 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.838557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.840382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.843742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:50.889780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1021/2056] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/projects => generated 393 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:50.893972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.908319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1036/2057] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:50.909757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578e2729-5f2b-427a-8087-ee751a0105ca None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:50.929718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578e2729-5f2b-427a-8087-ee751a0105ca None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.976961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3005cf5e-8d44-48c8-a0bf-91c1807fc331 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:50.987302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:50.988182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:50.988727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:50.989090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:50.990341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:50.993106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.001979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.002641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.002922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.003289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.003931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.006057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.014421 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1022/2058] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/users => generated 303 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:11:51.018311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.030410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b078ae-87f8-4eaf-ac8e-31867e927b65 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.036351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.037080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.037374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.037667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.038403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.040087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.048427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1037/2059] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:11:50 2022] POST /identity/v3/projects => generated 379 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:11:51.052092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.057009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1023/2060] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:11:51.059857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.070204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52228ff5-e552-4ed5-8eb3-d50a7e5d74c2 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.077584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:51.085480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:51.148453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80639b8-75c8-4240-955a-31297b2d0e39 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.155784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.156505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.156792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.157121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.158009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.159946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.166233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.167435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.167820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.168215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] PATH_INFO: `/v3/projects/63f38b050486475fb98b969487edf36b/users/307876fafa1d4dc1910b9c9b9d550afd/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.187597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63f38b050486475fb98b969487edf36b, user_id=307876fafa1d4dc1910b9c9b9d550afd, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.191001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.208607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1024/2061] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:51 2022] PUT /identity/v3/projects/63f38b050486475fb98b969487edf36b/users/307876fafa1d4dc1910b9c9b9d550afd/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:11:51.210698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.211386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1038/2062] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:11:51 2022] POST /identity/v3/users => generated 296 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:11:51.212374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.234153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7383898-cc3b-4a5c-9e21-40ca26824560 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.298169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42fac89-f745-4ba1-9a3e-bb6fef4e3010 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.303720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.304757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.305044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.305324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.306380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.307670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.321281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1025/2063] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:51.326312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.345600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d319ae9-ae53-4f86-9d93-9633b8401c48 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.348117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.349040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.349661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.350123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.351635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.354141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.361340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edaf1d75-b0db-41ff-8fc3-e9ff7950c6ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.362212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaf1d75-b0db-41ff-8fc3-e9ff7950c6ba admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.362495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaf1d75-b0db-41ff-8fc3-e9ff7950c6ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.362761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edaf1d75-b0db-41ff-8fc3-e9ff7950c6ba admin admin] PATH_INFO: `/v3/projects/c6f7f57b76e04c71b31608a8e6d35e63/users/34cc9c7b53564eb58370f20e5e80685a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.371455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1039/2064] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:51.375440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.388347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edaf1d75-b0db-41ff-8fc3-e9ff7950c6ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6f7f57b76e04c71b31608a8e6d35e63, user_id=34cc9c7b53564eb58370f20e5e80685a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.391054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a287f8ac-6f39-4f24-aefa-62d6a68eae1c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.391297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edaf1d75-b0db-41ff-8fc3-e9ff7950c6ba admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.395899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.396839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.397127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.397583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] PATH_INFO: `/v3/projects/63f38b050486475fb98b969487edf36b/users/307876fafa1d4dc1910b9c9b9d550afd/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.400016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63f38b050486475fb98b969487edf36b, user_id=307876fafa1d4dc1910b9c9b9d550afd, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.403397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.418073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1040/2065] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:51 2022] PUT /identity/v3/projects/63f38b050486475fb98b969487edf36b/users/307876fafa1d4dc1910b9c9b9d550afd/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:51.419824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1026/2066] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:51 2022] PUT /identity/v3/projects/c6f7f57b76e04c71b31608a8e6d35e63/users/34cc9c7b53564eb58370f20e5e80685a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:11:51.429602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ad218-40ae-49a5-9535-d045405b4024 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.430113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.430317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ad218-40ae-49a5-9535-d045405b4024 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.430740 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6ad218-40ae-49a5-9535-d045405b4024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.462144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fc6ad218-40ae-49a5-9535-d045405b4024 None None] MFA Rules not processed for user `307876fafa1d4dc1910b9c9b9d550afd`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:51.463394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fc6ad218-40ae-49a5-9535-d045405b4024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:51.501436 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0fc3d0-a645-4d29-a696-3f37031d6e43 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.542338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:51.546926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:51.566465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1027/2067] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:51 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:11:51.570349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.571093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.571376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.571638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.572566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.575035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.578383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6ad218-40ae-49a5-9535-d045405b4024 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.589638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1041/2068] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:11:51.590708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.600836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defef95a-0e69-4d33-b4e9-bf4388cc1e0a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.605964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f888ca5-285b-42fc-aa73-4c505bbd2fa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.606834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f888ca5-285b-42fc-aa73-4c505bbd2fa0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.607114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f888ca5-285b-42fc-aa73-4c505bbd2fa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.607392 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f888ca5-285b-42fc-aa73-4c505bbd2fa0 admin admin] PATH_INFO: `/v3/projects/c6f7f57b76e04c71b31608a8e6d35e63/users/34cc9c7b53564eb58370f20e5e80685a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.609406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f888ca5-285b-42fc-aa73-4c505bbd2fa0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6f7f57b76e04c71b31608a8e6d35e63, user_id=34cc9c7b53564eb58370f20e5e80685a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.612063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f888ca5-285b-42fc-aa73-4c505bbd2fa0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.623042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1042/2069] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:11:51 2022] PUT /identity/v3/projects/c6f7f57b76e04c71b31608a8e6d35e63/users/34cc9c7b53564eb58370f20e5e80685a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:11:51.634858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.634858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.634858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.638393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6ad218-40ae-49a5-9535-d045405b4024 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.661833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] MFA Rules not processed for user `34cc9c7b53564eb58370f20e5e80685a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:11:51.663055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:51.689376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.689955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.690237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.690513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.745264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1043/2070] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:11:51 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:11:51.758605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.760085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:11:51.761570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.763061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.771731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa7256a-f811-4669-a2b1-d0e497c7040d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.773368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1028/2071] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:51.775505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.776214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.776604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.777002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.786554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.787984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.798261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1044/2072] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:51.823984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.836322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b7f071-443b-40c5-8c9e-6835cab552e8 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.837552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:11:51.839060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.839763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.840052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.840336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.849200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c0febd-4993-445a-8070-7e4bc8e9c7c1 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:11:51.853283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564ea948-1ea9-4896-9b77-d65900dc5ab0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:11:51.853862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564ea948-1ea9-4896-9b77-d65900dc5ab0 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:11:51.854224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564ea948-1ea9-4896-9b77-d65900dc5ab0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:11:51.854495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564ea948-1ea9-4896-9b77-d65900dc5ab0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:11:51.908578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:11:51.911462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.913610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.923197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564ea948-1ea9-4896-9b77-d65900dc5ab0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:11:51.927756 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564ea948-1ea9-4896-9b77-d65900dc5ab0 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:11:51.930188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1029/2073] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:11:51.942874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1045/2074] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:11:51 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:01.765596 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:01.845957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970696c-91db-415e-b17e-482a9ad29ddf service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:01.851972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:01.853129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:01.853492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:01.853844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] PATH_INFO: `/v3/users/8be0abedb7b64f239ee7c9eb89263853` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:01.855352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8be0abedb7b64f239ee7c9eb89263853)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:01.858140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:01.913570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8be0abedb7b64f239ee7c9eb89263853'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:01.929623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8be0abedb7b64f239ee7c9eb89263853'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:01.932870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8be0abedb7b64f239ee7c9eb89263853'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:01.933660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8be0abedb7b64f239ee7c9eb89263853'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:01.938395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=86,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:01.948637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8be0abedb7b64f239ee7c9eb89263853'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:01.964254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1030/2075] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:01 2022] DELETE /identity/v3/users/8be0abedb7b64f239ee7c9eb89263853 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:01.969991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:01.970283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:01.970562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:01.989583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:01.991077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:02.065204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1046/2076] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:02.075389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:02.077645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:02.139233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7136a302-4b37-45c1-9bb1-42bceaa3a190 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:02.142907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:02.143560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:02.143844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:02.144123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:02.153735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:02.155094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:02.164532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1031/2077] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:02.474674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:02.476372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:02.506881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f0f872-da98-4eec-8de2-dc93c9a1e87c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:02.514738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:02.515705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:02.516064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:02.516446 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] PATH_INFO: `/v3/projects/cc5249e571e8469d9b1bd932cc02267b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:02.517867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc5249e571e8469d9b1bd932cc02267b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:02.521199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:02.612864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc5249e571e8469d9b1bd932cc02267b', 'request_id': 'req-b22be6c3-993d-406d-a719-68381fbe1e80'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.619261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc5249e571e8469d9b1bd932cc02267b', 'request_id': 'req-b22be6c3-993d-406d-a719-68381fbe1e80'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.629847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:02.631544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:02.631897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc5249e571e8469d9b1bd932cc02267b', 'request_id': 'req-b22be6c3-993d-406d-a719-68381fbe1e80'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.646862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] The token cache is being invalidate because project cc5249e571e8469d9b1bd932cc02267b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:02.647151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.647831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.651471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1047/2078] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:02 2022] DELETE /identity/v3/projects/cc5249e571e8469d9b1bd932cc02267b => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:02.653595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd99d7a-b853-4f88-8835-1e057d6f104d service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:02.656112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:02.660948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:02.670539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:02.741327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:02.742230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:02.742516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:02.742804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] PATH_INFO: `/v3/users/fadb4399d54543a6bfea88964a1bda0f` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:02.744012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fadb4399d54543a6bfea88964a1bda0f)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:02.746035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:02.754846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22be6c3-993d-406d-a719-68381fbe1e80 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:02.761467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:02.762461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:02.762819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:02.763166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] PATH_INFO: `/v3/users/307876fafa1d4dc1910b9c9b9d550afd` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:02.765524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=307876fafa1d4dc1910b9c9b9d550afd)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:02.767543 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:02.807769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:02.812612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fadb4399d54543a6bfea88964a1bda0f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.821249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fadb4399d54543a6bfea88964a1bda0f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.822389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fadb4399d54543a6bfea88964a1bda0f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.822931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fadb4399d54543a6bfea88964a1bda0f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.837497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8dcc4b21-bf89-4354-8475-b775174bc440 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fadb4399d54543a6bfea88964a1bda0f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.838713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '307876fafa1d4dc1910b9c9b9d550afd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.839379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '307876fafa1d4dc1910b9c9b9d550afd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.848264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '307876fafa1d4dc1910b9c9b9d550afd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.849279 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '307876fafa1d4dc1910b9c9b9d550afd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.856189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1032/2079] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:02 2022] DELETE /identity/v3/users/fadb4399d54543a6bfea88964a1bda0f => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:02.861902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '307876fafa1d4dc1910b9c9b9d550afd'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:02.862812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:02.865452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:02.865452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:02.892882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1048/2080] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:02 2022] DELETE /identity/v3/users/307876fafa1d4dc1910b9c9b9d550afd => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:02.895646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:02.896974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.012264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1033/2081] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:03.024046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.026600 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.121452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efdf2e9a-2736-478b-a208-ebd751b971b4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.129152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7e16ce-68a2-4164-bcb7-009778ca8674 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.129367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7e16ce-68a2-4164-bcb7-009778ca8674 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.129604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7e16ce-68a2-4164-bcb7-009778ca8674 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.129883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7e16ce-68a2-4164-bcb7-009778ca8674 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.141996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7e16ce-68a2-4164-bcb7-009778ca8674 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.143633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7e16ce-68a2-4164-bcb7-009778ca8674 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.169650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1049/2082] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:03.469972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.471475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.490621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709bd3f-3478-420c-9ffc-03cf66d5d2dd None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.498051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.499555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.500561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.501019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] PATH_INFO: `/v3/projects/63f38b050486475fb98b969487edf36b` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.502434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] RBAC: Authorizing `identity:delete_project(project_id=63f38b050486475fb98b969487edf36b)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.504573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.548729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:03.562966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63f38b050486475fb98b969487edf36b', 'request_id': 'req-93ab5473-052f-4724-bed0-324f9a486dbc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.568633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63f38b050486475fb98b969487edf36b', 'request_id': 'req-93ab5473-052f-4724-bed0-324f9a486dbc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.575711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63f38b050486475fb98b969487edf36b', 'request_id': 'req-93ab5473-052f-4724-bed0-324f9a486dbc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.588157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] The token cache is being invalidate because project 63f38b050486475fb98b969487edf36b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:03.588481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.589228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.590790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1034/2083] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:03 2022] DELETE /identity/v3/projects/63f38b050486475fb98b969487edf36b => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:03.595726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2267aa0b-9489-40be-9564-7c99ef323770 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.596024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2267aa0b-9489-40be-9564-7c99ef323770 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.596304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2267aa0b-9489-40be-9564-7c99ef323770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.614904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2267aa0b-9489-40be-9564-7c99ef323770 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:03.616113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2267aa0b-9489-40be-9564-7c99ef323770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.662827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.664369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.684614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1050/2084] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:03.692995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ae3c19-1157-41bd-be68-0cdf26c01057 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.693417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ae3c19-1157-41bd-be68-0cdf26c01057 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.693706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ae3c19-1157-41bd-be68-0cdf26c01057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.713984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-95ae3c19-1157-41bd-be68-0cdf26c01057 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:03.715248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-95ae3c19-1157-41bd-be68-0cdf26c01057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.723924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1051/2085] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:03.728508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ae3c19-1157-41bd-be68-0cdf26c01057 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.738355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ae3c19-1157-41bd-be68-0cdf26c01057 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.743170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.743876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.744166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.744298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ab5473-052f-4724-bed0-324f9a486dbc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.744481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.745209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:03.745630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.747957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.750258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.751122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.751422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.751677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] PATH_INFO: `/v3/projects/37d728b7da6b4b52aec0cdb5cb45bb79` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.752912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] RBAC: Authorizing `identity:delete_project(project_id=37d728b7da6b4b52aec0cdb5cb45bb79)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.754831 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.760829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1052/2086] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:03.764742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.774379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4466fb-a24b-4171-b387-55eaa43edc0f admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.779022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.779684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.779977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.780258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.780973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.782993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.807085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37d728b7da6b4b52aec0cdb5cb45bb79', 'request_id': 'req-fb093f32-4c2c-4772-897a-fb8c6372b60f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.812041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1053/2087] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:03 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:03.815955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.817065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37d728b7da6b4b52aec0cdb5cb45bb79', 'request_id': 'req-fb093f32-4c2c-4772-897a-fb8c6372b60f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.821667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:03.824116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37d728b7da6b4b52aec0cdb5cb45bb79', 'request_id': 'req-fb093f32-4c2c-4772-897a-fb8c6372b60f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.838195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] The token cache is being invalidate because project 37d728b7da6b4b52aec0cdb5cb45bb79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:03.838599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.839151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:03.840839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1035/2088] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:03 2022] DELETE /identity/v3/projects/37d728b7da6b4b52aec0cdb5cb45bb79 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:03.844660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.846011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.889540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8430b1f-86a7-498a-bfb6-c9c9bbcf9f22 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.894741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.895459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.895794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.896311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.897102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.900423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.903357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:03.919950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb093f32-4c2c-4772-897a-fb8c6372b60f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.924830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1054/2089] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:03 2022] POST /identity/v3/users => generated 284 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:03.929468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.929625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.930406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.930690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.931003 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] PATH_INFO: `/v3/users/d86a609b6c0f49c882c97f30cc479988` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.931341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:03.932266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d86a609b6c0f49c882c97f30cc479988)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.935812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.946057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82f6300-6ad7-4ae8-8f61-aeed34f014dd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.951232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.951763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.952047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.952373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.953419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.955170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.971116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1055/2090] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:03.971982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:03.983153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172747f7-6e0a-4602-8395-e1bc43710b2b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:03.988323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:03.989026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:03.989310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:03.989591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:03.990456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:03.992191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:03.999602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd86a609b6c0f49c882c97f30cc479988'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.006264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1056/2091] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:04.009753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd86a609b6c0f49c882c97f30cc479988'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.010609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd86a609b6c0f49c882c97f30cc479988'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.011468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd86a609b6c0f49c882c97f30cc479988'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.014193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.016355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:04.031192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3a35a4e5-2784-4602-89ad-fa8e3bcb2a80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd86a609b6c0f49c882c97f30cc479988'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.032977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c3ff7-20ff-441e-a9b7-6432fc2fbc8d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.038056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.038955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.039204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.039509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] PATH_INFO: `/v3/projects/c52bb9ed253a4c709e0d4f95ce6c1c28/users/086fd866de0d4361bdcf223bb736cd08/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.052306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1036/2092] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:03 2022] DELETE /identity/v3/users/d86a609b6c0f49c882c97f30cc479988 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:04.063700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] RBAC: Authorizing `identity:create_grant(project_id=c52bb9ed253a4c709e0d4f95ce6c1c28, user_id=086fd866de0d4361bdcf223bb736cd08, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.066655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.083324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1057/2093] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:04 2022] PUT /identity/v3/projects/c52bb9ed253a4c709e0d4f95ce6c1c28/users/086fd866de0d4361bdcf223bb736cd08/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:04.094118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc53c02-bf42-496f-a59d-6f2fd172d05a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.094492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc53c02-bf42-496f-a59d-6f2fd172d05a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.094877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc53c02-bf42-496f-a59d-6f2fd172d05a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.127123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-bbc53c02-bf42-496f-a59d-6f2fd172d05a None None] MFA Rules not processed for user `086fd866de0d4361bdcf223bb736cd08`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:04.129025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bbc53c02-bf42-496f-a59d-6f2fd172d05a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:04.237593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1037/2094] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:04 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:04.240643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.312427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94412acc-43bb-4c62-94df-ffdfd62a9ece admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.316859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.320499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.320845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.321133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.321859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.325641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.357292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1058/2095] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:04 2022] POST /identity/v3/projects => generated 365 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:04.358452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc53c02-bf42-496f-a59d-6f2fd172d05a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.369556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc53c02-bf42-496f-a59d-6f2fd172d05a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.437143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.437923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.438275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.438634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.439407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.441582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.450600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:04.472351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1038/2096] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:04 2022] POST /identity/v3/users => generated 289 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:04.473286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.479104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.484023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:04.490697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a70ed9-1ad9-4a5d-aa69-1fa7915664ba admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.495867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.497023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.497023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.497335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.497998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.499769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.525774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1059/2097] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:04 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:04.530676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.530825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b904040-0a0f-42d5-a6a2-a7fa62afcf02 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.544766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.545106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26947064-75bf-4445-867b-d8a524b508af admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.545815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.547223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.548520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] PATH_INFO: `/v3/projects/c55f817522494ce18875a76bb7b8d505` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.550336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=c55f817522494ce18875a76bb7b8d505)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.553159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.554365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.554928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.555217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] PATH_INFO: `/v3/projects/4ae9d73295134bacb951b0847429dc85/users/4a162def55044f1593500a93721a499a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.564629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.578317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ae9d73295134bacb951b0847429dc85, user_id=4a162def55044f1593500a93721a499a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.581418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.598206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:04.603473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1060/2098] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:04 2022] PUT /identity/v3/projects/4ae9d73295134bacb951b0847429dc85/users/4a162def55044f1593500a93721a499a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:04.607883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.642920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c55f817522494ce18875a76bb7b8d505', 'request_id': 'req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.649716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c55f817522494ce18875a76bb7b8d505', 'request_id': 'req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.658204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c55f817522494ce18875a76bb7b8d505', 'request_id': 'req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.682095 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] The token cache is being invalidate because project c55f817522494ce18875a76bb7b8d505 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:04.682622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.683288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.685170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1039/2099] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:04 2022] DELETE /identity/v3/projects/c55f817522494ce18875a76bb7b8d505 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:04.689162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.691187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:04.705375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0361f7-c10d-468d-acb6-90156eaacb85 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.794749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e13e5d8-e1ec-42e6-b1b1-20cea39a2a2b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.794989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.795640 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.795902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.796178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.796980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.798729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.801908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.802944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.803323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.803694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] PATH_INFO: `/v3/users/34cc9c7b53564eb58370f20e5e80685a` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.805521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34cc9c7b53564eb58370f20e5e80685a)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.807976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.813549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1061/2100] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:04 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:04.814469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:04.816180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:04.829288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb8f85-9a1a-4a0a-8772-8c773b330a8d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:04.834214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfa8513-2a7b-43b6-b77f-45c2b5854f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:04.835095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa8513-2a7b-43b6-b77f-45c2b5854f06 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.835382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa8513-2a7b-43b6-b77f-45c2b5854f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.835658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfa8513-2a7b-43b6-b77f-45c2b5854f06 admin admin] PATH_INFO: `/v3/projects/4ae9d73295134bacb951b0847429dc85/users/4a162def55044f1593500a93721a499a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.837773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfa8513-2a7b-43b6-b77f-45c2b5854f06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ae9d73295134bacb951b0847429dc85, user_id=4a162def55044f1593500a93721a499a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:04.840936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfa8513-2a7b-43b6-b77f-45c2b5854f06 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:04.857081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1062/2101] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:04 2022] PUT /identity/v3/projects/4ae9d73295134bacb951b0847429dc85/users/4a162def55044f1593500a93721a499a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:04.869077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:04.869560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:04.870184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:04.896678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34cc9c7b53564eb58370f20e5e80685a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.904360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34cc9c7b53564eb58370f20e5e80685a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.905123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34cc9c7b53564eb58370f20e5e80685a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.905767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34cc9c7b53564eb58370f20e5e80685a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.922339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34cc9c7b53564eb58370f20e5e80685a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:04.924525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:INSERT=4,keystone:SELECT=80,keystone:DELETE=16 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:04.929199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] MFA Rules not processed for user `4a162def55044f1593500a93721a499a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:04.931218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:04.951175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1040/2102] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:04 2022] DELETE /identity/v3/users/34cc9c7b53564eb58370f20e5e80685a => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:05.022378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1063/2103] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:04 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:05.033972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:05.035872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:05.098711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5689eebf-4e82-41ba-9b9d-cc4100cd31d9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:05.103311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:05.104033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:05.104486 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:05.106083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:05.173875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:05.175343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:05.186085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1041/2104] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:05 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:05.281647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:05.283346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:05.312823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:05.337239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:05.340065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:05.412963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ec377c-04c3-4c62-9ac2-b062a4ce430b None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:05.425796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22087852-8294-4869-9789-bcaf424e2e7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:05.426880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22087852-8294-4869-9789-bcaf424e2e7f service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:05.427316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22087852-8294-4869-9789-bcaf424e2e7f service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:05.427770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22087852-8294-4869-9789-bcaf424e2e7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:05.428812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c554c8c-744c-4a06-9192-37f48f5086f4 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:05.432918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:05.434537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:05.434835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:05.435132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] PATH_INFO: `/v3/projects/c6f7f57b76e04c71b31608a8e6d35e63` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:05.435855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-22087852-8294-4869-9789-bcaf424e2e7f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:05.436339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6f7f57b76e04c71b31608a8e6d35e63)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:05.440268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:05.531123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:INSERT=1,keystone:DELETE=4,keystone:SELECT=95 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:05.540506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6f7f57b76e04c71b31608a8e6d35e63', 'request_id': 'req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:05.562164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6f7f57b76e04c71b31608a8e6d35e63', 'request_id': 'req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:05.574288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6f7f57b76e04c71b31608a8e6d35e63', 'request_id': 'req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:05.575071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22087852-8294-4869-9789-bcaf424e2e7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:05.576981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22087852-8294-4869-9789-bcaf424e2e7f service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:05.600930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1064/2105] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:05 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:05.603673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] The token cache is being invalidate because project c6f7f57b76e04c71b31608a8e6d35e63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:05.604424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:05.606105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-33180b08-6a1a-4cd3-b7cc-3ea529af98d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:05.610088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1042/2106] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:05 2022] DELETE /identity/v3/projects/c6f7f57b76e04c71b31608a8e6d35e63 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:05.616238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1174d8b-d915-4f40-afa9-03b6777888bc None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:05.616599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1174d8b-d915-4f40-afa9-03b6777888bc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:05.616887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1174d8b-d915-4f40-afa9-03b6777888bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:05.644731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c1174d8b-d915-4f40-afa9-03b6777888bc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:05.646169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c1174d8b-d915-4f40-afa9-03b6777888bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:05.776476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1065/2107] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:05.780357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f418ad-3e64-40ad-9053-26421e7bea17 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:05.780705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f418ad-3e64-40ad-9053-26421e7bea17 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:05.780992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f418ad-3e64-40ad-9053-26421e7bea17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:05.819623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-98f418ad-3e64-40ad-9053-26421e7bea17 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:05.820946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-98f418ad-3e64-40ad-9053-26421e7bea17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:05.842673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1043/2108] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:05.847097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1174d8b-d915-4f40-afa9-03b6777888bc None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:05.866020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1174d8b-d915-4f40-afa9-03b6777888bc None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:05.873385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:05.874059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:05.874693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:05.875126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:05.876443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:05.876763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:05.880134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:05.903316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1066/2109] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:05.907521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f418ad-3e64-40ad-9053-26421e7bea17 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:05.917234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f418ad-3e64-40ad-9053-26421e7bea17 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:05.922181 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:05.922968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:05.923349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:05.924669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:05.934009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:05.937766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:05.985963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1044/2110] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:05 2022] POST /identity/v3/projects => generated 369 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:05.990020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:06.004599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be53843-094c-4b94-a8f2-9d084c5434e7 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:06.106468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:06.117812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:06.119316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.119993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.120881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.122059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:06.124936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:06.156424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1067/2111] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:05 2022] POST /identity/v3/users => generated 285 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:06.158210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:06.181760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c384e8db-dbf8-49f3-ac15-a38c6fea8f26 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:06.188870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:06.189741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.190190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.191698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.193036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:06.195039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:06.219032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1045/2112] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:06 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:06.223001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:06.232963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c11acd-1831-4a82-99df-3b274f33a3f9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:06.238239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:06.239218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.239602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.239884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.240974 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:06.242582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:06.254242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1068/2113] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:06 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:06.258599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:06.269956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9727b7f-ee9f-4724-83c1-b35a42225cc5 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:06.276287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c0d7b6-d9cd-4d8f-91fd-f1ada7f008cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:06.277221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0d7b6-d9cd-4d8f-91fd-f1ada7f008cb admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.277528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0d7b6-d9cd-4d8f-91fd-f1ada7f008cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.277807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0d7b6-d9cd-4d8f-91fd-f1ada7f008cb admin admin] PATH_INFO: `/v3/projects/0cc71645fab147959f329fe030163996/users/45838beec30e44b8ac45849c517dcd20/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.293075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:06.293798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c0d7b6-d9cd-4d8f-91fd-f1ada7f008cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=0cc71645fab147959f329fe030163996, user_id=45838beec30e44b8ac45849c517dcd20, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:06.295620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:06.296584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c0d7b6-d9cd-4d8f-91fd-f1ada7f008cb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:06.311842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3508dc99-7c70-4562-9e0f-b7ee11834584 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:06.314204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1046/2114] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:06 2022] PUT /identity/v3/projects/0cc71645fab147959f329fe030163996/users/45838beec30e44b8ac45849c517dcd20/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:06.318975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.319333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.319657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.339724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:06.342816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:06.386729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:06.387611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.387897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.388178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] PATH_INFO: `/v3/users/1e09f089d0a54814bf92eac518ad42f0` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.389439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e09f089d0a54814bf92eac518ad42f0)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:06.391418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:06.409998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1047/2115] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:06.420052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:06.421815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:06.441937 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e09f089d0a54814bf92eac518ad42f0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:06.442606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e09f089d0a54814bf92eac518ad42f0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:06.450378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e09f089d0a54814bf92eac518ad42f0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:06.451037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e09f089d0a54814bf92eac518ad42f0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:06.458827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4a1037ae-132e-4455-84aa-5bccba7028ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e09f089d0a54814bf92eac518ad42f0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:06.477977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1069/2116] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:06 2022] DELETE /identity/v3/users/1e09f089d0a54814bf92eac518ad42f0 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:06.482915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.483205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.483488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.493937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa66e624-d0be-4e56-b212-f5c82f43706a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:06.500990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:06.502371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:06.516063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:06.551300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:06.551961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.552257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.552601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.554529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:06.555007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:06.572674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1070/2117] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:06.581950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:06.583508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:06.596112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6970909-1850-4a73-a2b2-e60268ec7cce None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:06.600319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:06.601026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:06.601296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:06.601573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:06.613109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:06.614151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:06.627010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1071/2118] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:06.631481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:06.632953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:06.643327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1048/2119] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:07.137648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:07.139112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:07.164085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0209bd0-b2b4-4a9b-9982-c3e5416ef9aa service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:07.175785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:07.179705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:07.180034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:07.180327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] PATH_INFO: `/v3/projects/df29dee9940b489b85acbae532d9ad62` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:07.183476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] RBAC: Authorizing `identity:delete_project(project_id=df29dee9940b489b85acbae532d9ad62)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:07.190238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:07.268500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df29dee9940b489b85acbae532d9ad62', 'request_id': 'req-78cb9f88-be00-44e1-916e-95cfad8b3a2e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.280356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df29dee9940b489b85acbae532d9ad62', 'request_id': 'req-78cb9f88-be00-44e1-916e-95cfad8b3a2e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.292255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df29dee9940b489b85acbae532d9ad62', 'request_id': 'req-78cb9f88-be00-44e1-916e-95cfad8b3a2e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.306116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] The token cache is being invalidate because project df29dee9940b489b85acbae532d9ad62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:07.306777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.307181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-78cb9f88-be00-44e1-916e-95cfad8b3a2e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.309821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1072/2120] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:07 2022] DELETE /identity/v3/projects/df29dee9940b489b85acbae532d9ad62 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:07.313829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:07.325716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1d9f79-cc4d-4df0-b713-7e2e3f4f5585 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:07.470340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:07.471693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:07.472152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:07.472576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] PATH_INFO: `/v3/users/42492edb372d4887bc6a70bb3df6b0ed` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:07.474019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] RBAC: Authorizing `identity:delete_user(user_id=42492edb372d4887bc6a70bb3df6b0ed)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:07.476185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:07.561230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '42492edb372d4887bc6a70bb3df6b0ed'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.577945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '42492edb372d4887bc6a70bb3df6b0ed'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.579512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '42492edb372d4887bc6a70bb3df6b0ed'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.580272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '42492edb372d4887bc6a70bb3df6b0ed'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.601679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-e34ceb64-ef8b-40f1-957f-82ec9f47fb27 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '42492edb372d4887bc6a70bb3df6b0ed'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:07.606204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:07.634133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1049/2121] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:07 2022] DELETE /identity/v3/users/42492edb372d4887bc6a70bb3df6b0ed => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:07.783013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85820762-ec90-4cab-8122-24f594dee759 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:07.783752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85820762-ec90-4cab-8122-24f594dee759 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:07.784272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85820762-ec90-4cab-8122-24f594dee759 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:07.805416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-85820762-ec90-4cab-8122-24f594dee759 None None] MFA Rules not processed for user `d2d25be9c4674aa09b2975f1c6bc8c52`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:07.807276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-85820762-ec90-4cab-8122-24f594dee759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:07.880015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1073/2122] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:07 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:08.128522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.128975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.129906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.175087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] MFA Rules not processed for user `45838beec30e44b8ac45849c517dcd20`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:08.178547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:08.303406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85820762-ec90-4cab-8122-24f594dee759 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:08.305018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-85820762-ec90-4cab-8122-24f594dee759 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:08.343189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1050/2123] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:08 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:08.343189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:08.344688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:08.423488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85820762-ec90-4cab-8122-24f594dee759 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:08.440114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:08.441209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.443741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.444143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] PATH_INFO: `/v3/projects/18efaeb5acbd4def8945616c030aaa9c` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.448233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abe7818-f703-4723-ae87-e89fb89c8047 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:08.451049 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] RBAC: Authorizing `identity:delete_project(project_id=18efaeb5acbd4def8945616c030aaa9c)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:08.453271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:08.453657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:08.455116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.455504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.455881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.464845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:08.464845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:08.476496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:08.524359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1051/2124] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:08 2022] POST /identity/v3/projects => generated 371 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:08.536661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:08.562529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18efaeb5acbd4def8945616c030aaa9c', 'request_id': 'req-4d9ee3d6-3609-40a2-87b6-30a92354998e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:08.578742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18efaeb5acbd4def8945616c030aaa9c', 'request_id': 'req-4d9ee3d6-3609-40a2-87b6-30a92354998e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:08.591207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18efaeb5acbd4def8945616c030aaa9c', 'request_id': 'req-4d9ee3d6-3609-40a2-87b6-30a92354998e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:08.608457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] The token cache is being invalidate because project 18efaeb5acbd4def8945616c030aaa9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:08.608936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:08.609638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4d9ee3d6-3609-40a2-87b6-30a92354998e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:08.614572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1074/2125] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:08 2022] DELETE /identity/v3/projects/18efaeb5acbd4def8945616c030aaa9c => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:08.616671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917d2b19-faa8-47c7-b0dd-107abff72ae6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.617235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917d2b19-faa8-47c7-b0dd-107abff72ae6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.617519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917d2b19-faa8-47c7-b0dd-107abff72ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.639485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-917d2b19-faa8-47c7-b0dd-107abff72ae6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:08.641287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-917d2b19-faa8-47c7-b0dd-107abff72ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:08.701764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c0430e-c9ba-458a-b8af-2dff18ff1d57 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:08.741329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=1,keystone:SELECT=98,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:08.749717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1075/2126] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:08.765238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e100d107-7068-4727-aad9-fd79672046b3 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.765849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e100d107-7068-4727-aad9-fd79672046b3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.766211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e100d107-7068-4727-aad9-fd79672046b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.815811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e100d107-7068-4727-aad9-fd79672046b3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:08.817040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e100d107-7068-4727-aad9-fd79672046b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:08.819384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:08.820317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.820812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.821390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.822169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:08.824146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:08.845948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1076/2127] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:08.851700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e100d107-7068-4727-aad9-fd79672046b3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:08.862764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1052/2128] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:08 2022] POST /identity/v3/users => generated 292 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:08.869441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e100d107-7068-4727-aad9-fd79672046b3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:08.881755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:08.882445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.882736 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.883017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.883768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:08.884151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:08.885732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:08.889015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:08.890910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:08.922743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29449cd-0631-4bd8-a132-6cae761f2bd3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:08.931169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1077/2129] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:08.936679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:08.940873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:08.941663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.941956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.942230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.943025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:08.944995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:08.954442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbad7e5-8760-47ea-b422-75cdf7ad171e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:08.964432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:08.964432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:08.964877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:08.967809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:08.968798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:08.973006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:08.981688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1053/2130] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:08 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:08.983246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:08.997263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ad6780-db3a-4c86-9288-1d51379e5bec admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:09.030422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:09.031453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.032624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.033028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] PATH_INFO: `/v3/projects/14d94ce0ce5e486c80378acc134168f7/users/5b875d49366e4906937507e0b68292a3/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.047423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1078/2131] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:08 2022] POST /identity/v3/projects => generated 381 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:09.051632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:09.083321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14d94ce0ce5e486c80378acc134168f7, user_id=5b875d49366e4906937507e0b68292a3, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:09.085981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:09.106968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1054/2132] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:08 2022] PUT /identity/v3/projects/14d94ce0ce5e486c80378acc134168f7/users/5b875d49366e4906937507e0b68292a3/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:09.111243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:09.159966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:09.171742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661f1b6-6a9b-4a62-bcd6-ec33f863c6e6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:09.262551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4cceb-cc51-4084-b35a-3504b3a769c4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:09.270245 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:09.271526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.272959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.273627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.277545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:09.279481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:09.299916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1055/2133] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:09 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:09.304508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:09.314326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:09.314971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.315403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.315763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.316776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:09.317144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1d8dbc-febd-4a10-a14b-5b6b5c1043fb admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:09.319365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:09.326058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:09.326973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.327258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.327651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] PATH_INFO: `/v3/projects/14d94ce0ce5e486c80378acc134168f7/users/5b875d49366e4906937507e0b68292a3/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.329824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14d94ce0ce5e486c80378acc134168f7, user_id=5b875d49366e4906937507e0b68292a3, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:09.332886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:09.353338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1056/2134] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:09 2022] PUT /identity/v3/projects/14d94ce0ce5e486c80378acc134168f7/users/5b875d49366e4906937507e0b68292a3/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:09.354057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1079/2135] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:09 2022] POST /identity/v3/users => generated 291 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:09.356857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:09.492943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:09.555053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c2b812-e77f-413d-8f72-74d79bcbdd33 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:09.568647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:09.569905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.570343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.574379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.576925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:09.579295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:09.597731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1057/2136] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:09 2022] GET /identity/v3/roles => generated 1420 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:09.602580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:09.626267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ae8370-212d-4860-a428-db330a32318d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:09.637397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2928bc-0551-498c-9ba9-1ae53ed456bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:09.638388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2928bc-0551-498c-9ba9-1ae53ed456bd admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.638798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2928bc-0551-498c-9ba9-1ae53ed456bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.639337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2928bc-0551-498c-9ba9-1ae53ed456bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.640166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2928bc-0551-498c-9ba9-1ae53ed456bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:09.642044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2928bc-0551-498c-9ba9-1ae53ed456bd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:09.658247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1080/2137] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:09 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:09.664208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:09.687632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa66f64-10fb-406a-8c7b-1e36700b2c61 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:09.701371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:09.702294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.702584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.702867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] PATH_INFO: `/v3/projects/1b9877e6d95d4f6186b3f6caada2bffe/users/776442b60c234d27aba84485de392e10/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.719988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b9877e6d95d4f6186b3f6caada2bffe, user_id=776442b60c234d27aba84485de392e10, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:09.722841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:09.746083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1058/2138] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:09 2022] PUT /identity/v3/projects/1b9877e6d95d4f6186b3f6caada2bffe/users/776442b60c234d27aba84485de392e10/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:09.758111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a505c40-c414-4472-bbfb-1a70aaaaebc5 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:09.758407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a505c40-c414-4472-bbfb-1a70aaaaebc5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:09.760741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a505c40-c414-4472-bbfb-1a70aaaaebc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:09.811344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4a505c40-c414-4472-bbfb-1a70aaaaebc5 None None] MFA Rules not processed for user `776442b60c234d27aba84485de392e10`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:09.812828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4a505c40-c414-4472-bbfb-1a70aaaaebc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:09.971769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1081/2139] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:09 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:09.977519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:10.083153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4082d40a-1835-4035-8d38-4ef425df0049 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:10.090135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:10.091049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.091594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.092163 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.093388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:10.095878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:10.138624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1059/2140] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:09 2022] POST /identity/v3/projects => generated 381 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:10.142619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a505c40-c414-4472-bbfb-1a70aaaaebc5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:10.156896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a505c40-c414-4472-bbfb-1a70aaaaebc5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:10.223401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:10.224589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.225010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.225431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.226308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:10.228311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:10.266576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1082/2141] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:10 2022] POST /identity/v3/users => generated 297 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:10.270193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:10.287640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822e5413-9806-41ff-a2c1-3a3c89f1628e admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:10.297344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:10.298354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.298837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.299270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.300240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:10.305121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:10.327124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1060/2142] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:10 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:10.331416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:10.337421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:10.344452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0231a73f-5ff0-41f2-a4c1-051c44d2cd55 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:10.351922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:10.353188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.353661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.354081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] PATH_INFO: `/v3/projects/21eb356c7553446086a3a0fe6d4ed29a/users/6cc9a1e686a64ca296c77a68b4897be7/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.399112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21eb356c7553446086a3a0fe6d4ed29a, user_id=6cc9a1e686a64ca296c77a68b4897be7, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:10.404236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:10.470067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1083/2143] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:10 2022] PUT /identity/v3/projects/21eb356c7553446086a3a0fe6d4ed29a/users/6cc9a1e686a64ca296c77a68b4897be7/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:10.471401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:10.486141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84989b59-c4a7-47a0-835e-a94434cfcc20 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:10.590395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6cb5a2-d8a7-4846-87df-81f031494ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:10.591480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6cb5a2-d8a7-4846-87df-81f031494ed0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.592729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6cb5a2-d8a7-4846-87df-81f031494ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.593165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6cb5a2-d8a7-4846-87df-81f031494ed0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.594083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6cb5a2-d8a7-4846-87df-81f031494ed0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:10.595874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6cb5a2-d8a7-4846-87df-81f031494ed0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:10.611722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1061/2144] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:10 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:10.616220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:10.630274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa8af16-b4f9-4a20-986c-8081741f2217 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:10.656280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ffe1c9-dca3-4d08-a080-09bba9790d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:10.662655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ffe1c9-dca3-4d08-a080-09bba9790d2f admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.662834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ffe1c9-dca3-4d08-a080-09bba9790d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.663121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ffe1c9-dca3-4d08-a080-09bba9790d2f admin admin] PATH_INFO: `/v3/projects/21eb356c7553446086a3a0fe6d4ed29a/users/6cc9a1e686a64ca296c77a68b4897be7/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.666037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ffe1c9-dca3-4d08-a080-09bba9790d2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=21eb356c7553446086a3a0fe6d4ed29a, user_id=6cc9a1e686a64ca296c77a68b4897be7, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:10.671878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ffe1c9-dca3-4d08-a080-09bba9790d2f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:10.691688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1084/2145] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:10 2022] PUT /identity/v3/projects/21eb356c7553446086a3a0fe6d4ed29a/users/6cc9a1e686a64ca296c77a68b4897be7/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:10.704375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.704932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.705305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.765400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] MFA Rules not processed for user `6cc9a1e686a64ca296c77a68b4897be7`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:10.767532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:10.831633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:10.885216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71910c7f-009d-4f69-920e-204a02079c8e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:10.885431 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71910c7f-009d-4f69-920e-204a02079c8e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:10.885702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71910c7f-009d-4f69-920e-204a02079c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:10.917949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1062/2146] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:10 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:10.942974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:10.951136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:10.986896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-71910c7f-009d-4f69-920e-204a02079c8e None None] MFA Rules not processed for user `5b875d49366e4906937507e0b68292a3`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:10.989324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-71910c7f-009d-4f69-920e-204a02079c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:11.025570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bacb0ea-0a8c-4908-be24-8331f44c2d26 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:11.030220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:11.031166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:11.031458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:11.031738 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:11.041933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:11.043953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:11.056030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1063/2147] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:10 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:11.112091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1085/2148] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:10 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:11.124068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:11.126678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:11.144752 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1db218-7d0c-4c06-aa41-6b2a88cb1595 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:11.150080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:11.150753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:11.151043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:11.151336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:11.154280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:11.277027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:11.278634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:11.301049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1064/2149] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:11 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:11.881321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71910c7f-009d-4f69-920e-204a02079c8e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:11.900378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71910c7f-009d-4f69-920e-204a02079c8e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:11.911770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cc9a1e686a64ca296c77a68b4897be7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '21eb356c7553446086a3a0fe6d4ed29a', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:11.912491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:11.912802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:11.913078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:11.919876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:11.920406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:11.923497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:11.931511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:11.933084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:11.961940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1086/2150] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:11.963811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:11.983353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1977f11-d3ee-43f7-95f2-239b9b2db678 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:11.988305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cc9a1e686a64ca296c77a68b4897be7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '21eb356c7553446086a3a0fe6d4ed29a', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:11.989189 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:11.989613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:11.990049 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:11.990744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:11.992676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:12.015612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149c43f6-1aab-4226-a5c3-b0dd0252f8b4 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:12.021068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:12.022026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:12.022305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:12.022575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] PATH_INFO: `/v3/users/086fd866de0d4361bdcf223bb736cd08` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:12.023958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] RBAC: Authorizing `identity:delete_user(user_id=086fd866de0d4361bdcf223bb736cd08)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:12.025968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:12.028315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1087/2151] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:11 2022] POST /identity/v3/projects => generated 354 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:12.032534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:12.033064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:12.092269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '086fd866de0d4361bdcf223bb736cd08'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.104485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '086fd866de0d4361bdcf223bb736cd08'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.105190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '086fd866de0d4361bdcf223bb736cd08'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.106352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '086fd866de0d4361bdcf223bb736cd08'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.112880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a97c22-f48b-419d-8839-c4ba86a982a8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:12.118928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cc9a1e686a64ca296c77a68b4897be7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '21eb356c7553446086a3a0fe6d4ed29a', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:12.120755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:12.121300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:12.121433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:12.122239 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:12.122429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3500b46-d5f6-4426-bf41-bb8364f51179 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '086fd866de0d4361bdcf223bb736cd08'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.122621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:12.124185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:12.141146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1065/2152] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:11 2022] DELETE /identity/v3/users/086fd866de0d4361bdcf223bb736cd08 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:12.144679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1088/2153] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:12.145661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:12.145960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:12.146818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:12.169976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:12.171744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:12.207101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:12.209388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:12.227733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f47cea2-c768-4f62-b957-1ae4c32ce8c8 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:12.233767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:12.283901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1066/2154] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:12.285885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:12.287326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:12.330597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6cc9a1e686a64ca296c77a68b4897be7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '21eb356c7553446086a3a0fe6d4ed29a', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:12.332231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:12.332231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:12.332497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] PATH_INFO: `/v3/projects/e1b22546eaec41ffa294863e81e61c68` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:12.334065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorizing `identity:delete_project(project_id=e1b22546eaec41ffa294863e81e61c68)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:12.336250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:12.362324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241fabd6-1621-47c1-8601-2fd3954986be None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:12.366670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:12.367334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:12.367985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:12.367985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:12.380104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:12.382614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:12.406547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1067/2155] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:12.431185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1b22546eaec41ffa294863e81e61c68', 'request_id': 'req-c74ed49e-5222-44a9-9ce8-c90228715d42'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.447553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1b22546eaec41ffa294863e81e61c68', 'request_id': 'req-c74ed49e-5222-44a9-9ce8-c90228715d42'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.462428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1b22546eaec41ffa294863e81e61c68', 'request_id': 'req-c74ed49e-5222-44a9-9ce8-c90228715d42'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.483739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] The token cache is being invalidate because project e1b22546eaec41ffa294863e81e61c68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:12.484134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.485153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:12.487081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1089/2156] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:12 2022] DELETE /identity/v3/projects/e1b22546eaec41ffa294863e81e61c68 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:12.868857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:12.870613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:12.898619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039c3d70-3bcf-4530-bfd4-97f095dfb947 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:12.988289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:12.993771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:12.999455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:13.108578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:13.109736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.110307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.110591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] PATH_INFO: `/v3/projects/c52bb9ed253a4c709e0d4f95ce6c1c28` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.112430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=c52bb9ed253a4c709e0d4f95ce6c1c28)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:13.114386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:13.141328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ed49e-5222-44a9-9ce8-c90228715d42 tempest-VolumeQuotaClassesTest-1082937998 tempest-VolumeQuotaClassesTest-1082937998-project-admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:13.155695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:13.156361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.156727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.157009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.158862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.217737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:13.219540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c52bb9ed253a4c709e0d4f95ce6c1c28', 'request_id': 'req-391b4c73-328c-4f6a-8985-07d26d9209ae'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.234796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:13.236241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:13.240281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c52bb9ed253a4c709e0d4f95ce6c1c28', 'request_id': 'req-391b4c73-328c-4f6a-8985-07d26d9209ae'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.249146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1090/2157] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:12 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:13.249713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c52bb9ed253a4c709e0d4f95ce6c1c28', 'request_id': 'req-391b4c73-328c-4f6a-8985-07d26d9209ae'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.250010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:13.251420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.272630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59620d01-2b87-40fd-8860-ad6be6144fe5 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:13.276495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] The token cache is being invalidate because project c52bb9ed253a4c709e0d4f95ce6c1c28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:13.277337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.278793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.281484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1068/2158] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:12 2022] DELETE /identity/v3/projects/c52bb9ed253a4c709e0d4f95ce6c1c28 => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:13.285347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:13.286839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.403344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391b4c73-328c-4f6a-8985-07d26d9209ae admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:13.405969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:13.407394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.407939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.408780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] PATH_INFO: `/v3/users/776442b60c234d27aba84485de392e10` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.411182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=776442b60c234d27aba84485de392e10)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:13.417026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:13.417162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:13.418050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.418396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.420495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] PATH_INFO: `/v3/users/4a162def55044f1593500a93721a499a` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.422431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a162def55044f1593500a93721a499a)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:13.427444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:13.512585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '776442b60c234d27aba84485de392e10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.514859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '776442b60c234d27aba84485de392e10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.520725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a162def55044f1593500a93721a499a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.529962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a162def55044f1593500a93721a499a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.530792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a162def55044f1593500a93721a499a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.531479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a162def55044f1593500a93721a499a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.532911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:13.536684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '776442b60c234d27aba84485de392e10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.538215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '776442b60c234d27aba84485de392e10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.548752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-fad2d642-0ceb-42a3-b073-a6d42427da1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '776442b60c234d27aba84485de392e10'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.551457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a162def55044f1593500a93721a499a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.577932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1091/2159] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:13 2022] DELETE /identity/v3/users/776442b60c234d27aba84485de392e10 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:13.582277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ae84e-1cda-45cf-b7d1-52d5eb963868 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.582277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ae84e-1cda-45cf-b7d1-52d5eb963868 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.582277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ae84e-1cda-45cf-b7d1-52d5eb963868 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.583543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1069/2160] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:13 2022] DELETE /identity/v3/users/4a162def55044f1593500a93721a499a => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:13.584662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:13.591503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.593015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1092/2161] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:12:13 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:12:13.597161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db94357d-ed7c-4af8-bef7-b9fe56f6846c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.597517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db94357d-ed7c-4af8-bef7-b9fe56f6846c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.598047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db94357d-ed7c-4af8-bef7-b9fe56f6846c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.648261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-db94357d-ed7c-4af8-bef7-b9fe56f6846c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:13.649982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-db94357d-ed7c-4af8-bef7-b9fe56f6846c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.693947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8611914-8158-4f8d-85a4-82d60225c162 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:13.703127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:13.704575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.704968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.705833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] PATH_INFO: `/v3/users/a9d95652b0714a2a8427a7e99de27bee` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.707516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9d95652b0714a2a8427a7e99de27bee)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:13.710744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:13.757211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1093/2162] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:13.758857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.759202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.759524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.783777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] MFA Rules not processed for user `947da12bc8304314b5e4a9768b083a7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:13.785043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.791651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:13.793535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9d95652b0714a2a8427a7e99de27bee'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.806481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9d95652b0714a2a8427a7e99de27bee'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.807397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9d95652b0714a2a8427a7e99de27bee'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.810142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9d95652b0714a2a8427a7e99de27bee'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.836875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9d95652b0714a2a8427a7e99de27bee'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:13.860597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1070/2163] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:13 2022] DELETE /identity/v3/users/a9d95652b0714a2a8427a7e99de27bee => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:13.861700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:13.863247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.889933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1094/2164] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:12:13 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:13.931521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:13.933409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.947240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188929a4-2077-4ab1-b983-8de22fea3280 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:13.950540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341372cb-0786-4968-9f1a-c8d58d55e344 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:13.951618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:13.952234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:13.952556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:13.952839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:13.955383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:13.963311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:14.048015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d9c16-a5b2-489d-8a53-b22bfe82ef1d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.048857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d9c16-a5b2-489d-8a53-b22bfe82ef1d service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.049037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d9c16-a5b2-489d-8a53-b22bfe82ef1d service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.049319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d9c16-a5b2-489d-8a53-b22bfe82ef1d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.051640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.053221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.060792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d9c16-a5b2-489d-8a53-b22bfe82ef1d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.062986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d9c16-a5b2-489d-8a53-b22bfe82ef1d service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.069959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1071/2165] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:14.088212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1095/2166] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:13 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:14.149633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.151213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.165239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85372e4-e8fe-4df4-8732-e9b07a6d5477 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.171161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.172234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.172562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.173413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] PATH_INFO: `/v3/projects/f1f4fee9c6b844a2b7a14e2afb831f88` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.175323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1f4fee9c6b844a2b7a14e2afb831f88)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.177455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.264613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1f4fee9c6b844a2b7a14e2afb831f88', 'request_id': 'req-30ef8fa4-f92b-4c46-be31-9de1f988fa85'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.274560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1f4fee9c6b844a2b7a14e2afb831f88', 'request_id': 'req-30ef8fa4-f92b-4c46-be31-9de1f988fa85'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.281558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1f4fee9c6b844a2b7a14e2afb831f88', 'request_id': 'req-30ef8fa4-f92b-4c46-be31-9de1f988fa85'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.297424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] The token cache is being invalidate because project f1f4fee9c6b844a2b7a14e2afb831f88 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:14.298211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.298369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-30ef8fa4-f92b-4c46-be31-9de1f988fa85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.300785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1072/2167] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:14 2022] DELETE /identity/v3/projects/f1f4fee9c6b844a2b7a14e2afb831f88 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:14.305023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a008e0-ba2f-4470-9853-77a530304557 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.305316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a008e0-ba2f-4470-9853-77a530304557 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.305624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a008e0-ba2f-4470-9853-77a530304557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.324801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-83a008e0-ba2f-4470-9853-77a530304557 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:14.326004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-83a008e0-ba2f-4470-9853-77a530304557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.406523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1096/2168] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:14.414403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.414867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.415553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.440501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:14.442080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.452742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1073/2169] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:14.457148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a008e0-ba2f-4470-9853-77a530304557 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.470726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a008e0-ba2f-4470-9853-77a530304557 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.476526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.477191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.477465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.477749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.478558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:14.478805 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.480451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.494340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.498046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.509072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6149153f-2c39-4d42-9596-308992cf5c07 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.510240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1097/2170] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:14.514498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.514734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.515638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.515929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.516208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] PATH_INFO: `/v3/projects/1b9877e6d95d4f6186b3f6caada2bffe` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.517488 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b9877e6d95d4f6186b3f6caada2bffe)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.519484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.537334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a57d3ab-2d66-486d-a1d2-c93898b9f4f9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.544798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.545481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.545819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.546113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.546753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.548758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.574564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:14.579924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b9877e6d95d4f6186b3f6caada2bffe', 'request_id': 'req-143304ed-6f11-4c73-b7c4-f67378842fad'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.583782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1098/2171] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:14 2022] POST /identity/v3/projects => generated 369 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:14.585355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.587127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.590121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b9877e6d95d4f6186b3f6caada2bffe', 'request_id': 'req-143304ed-6f11-4c73-b7c4-f67378842fad'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.599373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b9877e6d95d4f6186b3f6caada2bffe', 'request_id': 'req-143304ed-6f11-4c73-b7c4-f67378842fad'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.621224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] The token cache is being invalidate because project 1b9877e6d95d4f6186b3f6caada2bffe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:14.621998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.622864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.625574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1074/2172] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:14 2022] DELETE /identity/v3/projects/1b9877e6d95d4f6186b3f6caada2bffe => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:14.631946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.634050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.641221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:14.725287 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143304ed-6f11-4c73-b7c4-f67378842fad admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.729086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8c8e43-c777-4664-a5ef-b8dd3e521f44 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.744040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.745571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.745868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.746693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.749057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.749306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.750387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.750677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.751519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] PATH_INFO: `/v3/projects/4ae9d73295134bacb951b0847429dc85` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.753654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ae9d73295134bacb951b0847429dc85)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.756335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.758130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.796153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1075/2173] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:14 2022] POST /identity/v3/users => generated 285 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:14.797127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.798793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.813379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eddd4bb-ed88-4f90-8a48-cc712d29d926 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.830670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ae9d73295134bacb951b0847429dc85', 'request_id': 'req-7760bd96-48f0-40e1-b1a4-2d77a559fe05'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.846468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ae9d73295134bacb951b0847429dc85', 'request_id': 'req-7760bd96-48f0-40e1-b1a4-2d77a559fe05'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.856930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ae9d73295134bacb951b0847429dc85', 'request_id': 'req-7760bd96-48f0-40e1-b1a4-2d77a559fe05'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.876043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] The token cache is being invalidate because project 4ae9d73295134bacb951b0847429dc85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:14.876993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.877677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.880444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.880585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1099/2174] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:14 2022] DELETE /identity/v3/projects/4ae9d73295134bacb951b0847429dc85 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:14.881378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.881641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.881919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] PATH_INFO: `/v3/users/6cc9a1e686a64ca296c77a68b4897be7` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.882574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.883318 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cc9a1e686a64ca296c77a68b4897be7)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.884118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.885649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.901315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7760bd96-48f0-40e1-b1a4-2d77a559fe05 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.909684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b06092d-731d-499c-9548-1ad836fa0db4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.910367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b06092d-731d-499c-9548-1ad836fa0db4 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.910649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b06092d-731d-499c-9548-1ad836fa0db4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.910923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b06092d-731d-499c-9548-1ad836fa0db4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.911701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b06092d-731d-499c-9548-1ad836fa0db4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.913311 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b06092d-731d-499c-9548-1ad836fa0db4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:14.924527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1100/2175] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:14.926103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00caafad-561f-48e3-bac9-9123c31b8494 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.926395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00caafad-561f-48e3-bac9-9123c31b8494 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.926677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00caafad-561f-48e3-bac9-9123c31b8494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.964263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-00caafad-561f-48e3-bac9-9123c31b8494 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:14.966132 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-00caafad-561f-48e3-bac9-9123c31b8494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:14.968702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cc9a1e686a64ca296c77a68b4897be7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.975567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1101/2176] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:14.976429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00caafad-561f-48e3-bac9-9123c31b8494 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:14.978707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cc9a1e686a64ca296c77a68b4897be7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.979484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cc9a1e686a64ca296c77a68b4897be7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.980132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cc9a1e686a64ca296c77a68b4897be7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.986738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00caafad-561f-48e3-bac9-9123c31b8494 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:14.994489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cc9a1e686a64ca296c77a68b4897be7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:14.995204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f33005-84e4-4814-b26f-3b39fc9de4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:14.996127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f33005-84e4-4814-b26f-3b39fc9de4bc admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:14.996617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f33005-84e4-4814-b26f-3b39fc9de4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:14.996899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f33005-84e4-4814-b26f-3b39fc9de4bc admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:14.997815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f33005-84e4-4814-b26f-3b39fc9de4bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:14.999994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f33005-84e4-4814-b26f-3b39fc9de4bc admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.013746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1102/2177] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:14 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:15.014397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1076/2178] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:14 2022] DELETE /identity/v3/users/6cc9a1e686a64ca296c77a68b4897be7 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:15.018376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f03ca-e0a8-45ac-8884-643f622a70dc None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.018494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.018675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f03ca-e0a8-45ac-8884-643f622a70dc None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.018965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3f03ca-e0a8-45ac-8884-643f622a70dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.020791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:15.036172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:15.053719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8b3f03ca-e0a8-45ac-8884-643f622a70dc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:15.055095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8b3f03ca-e0a8-45ac-8884-643f622a70dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:15.112948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6384ce35-0e10-4969-86cf-cee997e5c484 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.123679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbcead-e0f0-434e-94aa-9d66bd86fa69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.124977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbcead-e0f0-434e-94aa-9d66bd86fa69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.125495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbcead-e0f0-434e-94aa-9d66bd86fa69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.126227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbcead-e0f0-434e-94aa-9d66bd86fa69 admin admin] PATH_INFO: `/v3/projects/a6f1f3d7ffff4273a18637cf0fb73cbc/users/377f96c18d264057aec1dd7ea4217072/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.147812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbbcead-e0f0-434e-94aa-9d66bd86fa69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a6f1f3d7ffff4273a18637cf0fb73cbc, user_id=377f96c18d264057aec1dd7ea4217072, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.150715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbbcead-e0f0-434e-94aa-9d66bd86fa69 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.151654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1103/2179] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:15.157579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3f03ca-e0a8-45ac-8884-643f622a70dc None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.163378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:15.169408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1077/2180] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:15 2022] PUT /identity/v3/projects/a6f1f3d7ffff4273a18637cf0fb73cbc/users/377f96c18d264057aec1dd7ea4217072/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:15.170429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3f03ca-e0a8-45ac-8884-643f622a70dc None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.179172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8105d6ba-b6bb-4bae-930c-13ddaf1e66b1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.179457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8105d6ba-b6bb-4bae-930c-13ddaf1e66b1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.179749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8105d6ba-b6bb-4bae-930c-13ddaf1e66b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.210806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8105d6ba-b6bb-4bae-930c-13ddaf1e66b1 None None] MFA Rules not processed for user `377f96c18d264057aec1dd7ea4217072`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:15.212164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8105d6ba-b6bb-4bae-930c-13ddaf1e66b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:15.257043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.258169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.258587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.258818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.259837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:15.260822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.263916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.287081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1104/2181] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:15.288713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.298969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d7491a-6d1b-43e8-9a54-34c783d39c19 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.304115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.304837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.305123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.305490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.306103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.308160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1078/2182] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:15.308277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.312762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8105d6ba-b6bb-4bae-930c-13ddaf1e66b1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.328001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8105d6ba-b6bb-4bae-930c-13ddaf1e66b1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.333551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.334216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.334497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.334796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.335574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.337611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.342610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1105/2183] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:15.343582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.367900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1079/2184] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/projects => generated 369 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:15.372200 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.420928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971c54be-f93c-4e5e-b69d-602bc7a3d592 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.468351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab43abf-247f-42fe-8aa7-4a6d24d039ee admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.480528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.481233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.481518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.481797 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.482490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.484565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.492883 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:15.500314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.501303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.501786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.502275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.503287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.506588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.515017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1080/2185] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/users => generated 291 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:15.516209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.528418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b51cf-3788-4c69-8a86-41962fb9c247 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.533527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.534197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.534491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.534804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.535550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.537408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.540413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1106/2186] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/users => generated 291 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:15.544309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.549519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1081/2187] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:15 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:15.553416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.564161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9489fe6-ff2b-47b2-ac91-03d93bea32ad admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.564903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c884e182-cfd8-4cce-b5e7-b9c93a982528 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.570241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.570589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.571137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.571321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.571420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.571588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.571696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] PATH_INFO: `/v3/projects/47b8c0d424dd4f4abeef3fa4b2c327ed/users/e6e5d04710784b50b72dabd85b50e123/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.571874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.572931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.576462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.587768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:15.593545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47b8c0d424dd4f4abeef3fa4b2c327ed, user_id=e6e5d04710784b50b72dabd85b50e123, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.596760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.596983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1107/2188] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:15 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:15.599707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.612427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a748df74-3d76-4f5e-959a-fb5d6892ecbf admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.613699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1082/2189] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:15 2022] PUT /identity/v3/projects/47b8c0d424dd4f4abeef3fa4b2c327ed/users/e6e5d04710784b50b72dabd85b50e123/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:15.618232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.696415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.697359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.697670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.697963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.698803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.700882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.703054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3594919b-540e-412e-9378-91f110c811f3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.709994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.710690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.710983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.711267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.712017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.714045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.714612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1108/2190] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:15 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:15.715544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.717653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:15.737737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1083/2191] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:15 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:15.737963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3a7b6b-4214-4c1f-babd-f49fd4bb870a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.738069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.751833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b69322c-d7cc-4774-8252-4d9c8a9e9d0f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.754962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.755855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.756146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.756454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] PATH_INFO: `/v3/projects/21eb356c7553446086a3a0fe6d4ed29a` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.757750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21eb356c7553446086a3a0fe6d4ed29a)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.759703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.759882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.761626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.762627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.763098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] PATH_INFO: `/v3/projects/c361030bed824e3290f7769e28c6f4af/users/b8354a0625894297b21d10d5031d900f/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.782142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] RBAC: Authorizing `identity:create_grant(project_id=c361030bed824e3290f7769e28c6f4af, user_id=b8354a0625894297b21d10d5031d900f, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.786604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.817680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1084/2192] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:15 2022] PUT /identity/v3/projects/c361030bed824e3290f7769e28c6f4af/users/b8354a0625894297b21d10d5031d900f/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:15.817680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:15.817985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21eb356c7553446086a3a0fe6d4ed29a', 'request_id': 'req-c7bda4ac-3bda-497f-9af7-eda912e941c1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:15.824338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21eb356c7553446086a3a0fe6d4ed29a', 'request_id': 'req-c7bda4ac-3bda-497f-9af7-eda912e941c1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:15.834356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21eb356c7553446086a3a0fe6d4ed29a', 'request_id': 'req-c7bda4ac-3bda-497f-9af7-eda912e941c1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:15.850718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] The token cache is being invalidate because project 21eb356c7553446086a3a0fe6d4ed29a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:15.850949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:15.851575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7bda4ac-3bda-497f-9af7-eda912e941c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:15.853779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1109/2193] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:15 2022] DELETE /identity/v3/projects/21eb356c7553446086a3a0fe6d4ed29a => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:15.855089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f5cb85-d50f-4023-9c50-39a24e121751 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.855475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f5cb85-d50f-4023-9c50-39a24e121751 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.855785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f5cb85-d50f-4023-9c50-39a24e121751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.884963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-83f5cb85-d50f-4023-9c50-39a24e121751 None None] MFA Rules not processed for user `b8354a0625894297b21d10d5031d900f`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:15.886495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-83f5cb85-d50f-4023-9c50-39a24e121751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:15.890304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7404b5a4-dadb-45e0-8359-9cce63ba7eef admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:15.942332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:15.967040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1110/2194] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:15.968697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ac6e7-cda5-4307-9835-3d4b8779fe64 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.969177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ac6e7-cda5-4307-9835-3d4b8779fe64 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.969459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ac6e7-cda5-4307-9835-3d4b8779fe64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.978100 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:15.979021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:15.979308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:15.979593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] PATH_INFO: `/v3/projects/47b8c0d424dd4f4abeef3fa4b2c327ed/users/e6e5d04710784b50b72dabd85b50e123/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:15.982209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=47b8c0d424dd4f4abeef3fa4b2c327ed, user_id=e6e5d04710784b50b72dabd85b50e123, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:15.985743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:15.989413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7a7ac6e7-cda5-4307-9835-3d4b8779fe64 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:15.990592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7a7ac6e7-cda5-4307-9835-3d4b8779fe64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:15.994959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:15.999349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1085/2195] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:15 2022] PUT /identity/v3/projects/47b8c0d424dd4f4abeef3fa4b2c327ed/users/e6e5d04710784b50b72dabd85b50e123/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:16.003757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.006607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:16.076659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1111/2196] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:16.078382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6af4765-445c-4693-b0eb-e720a3fa52ca None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.078706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6af4765-445c-4693-b0eb-e720a3fa52ca None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.079216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6af4765-445c-4693-b0eb-e720a3fa52ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.099104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12b4e4-3f48-4218-a494-2835596ac8e9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.109099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa68146-8b96-46ff-9e57-ad2b82698e2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.110345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa68146-8b96-46ff-9e57-ad2b82698e2f admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.111748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa68146-8b96-46ff-9e57-ad2b82698e2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.112212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa68146-8b96-46ff-9e57-ad2b82698e2f admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.113281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa68146-8b96-46ff-9e57-ad2b82698e2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.116015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa68146-8b96-46ff-9e57-ad2b82698e2f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.123622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c6af4765-445c-4693-b0eb-e720a3fa52ca None None] MFA Rules not processed for user `e6e5d04710784b50b72dabd85b50e123`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:16.125086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c6af4765-445c-4693-b0eb-e720a3fa52ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:16.163796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1086/2197] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:16 2022] POST /identity/v3/projects => generated 381 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:16.165432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0035164-e64a-4265-9519-a358704a8303 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.165727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0035164-e64a-4265-9519-a358704a8303 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.166009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0035164-e64a-4265-9519-a358704a8303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.189999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d0035164-e64a-4265-9519-a358704a8303 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:16.191191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d0035164-e64a-4265-9519-a358704a8303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:16.207247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1112/2198] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:16 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:16.207788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6af4765-445c-4693-b0eb-e720a3fa52ca None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.230150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6af4765-445c-4693-b0eb-e720a3fa52ca None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.274910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1087/2199] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:16.275783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0035164-e64a-4265-9519-a358704a8303 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.277113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d0035164-e64a-4265-9519-a358704a8303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:16.318296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.319071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.319447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.319815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.320843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.322740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.346690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1113/2200] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:16 2022] POST /identity/v3/users => generated 297 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:16.347408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.357662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fef600-046e-4708-bfda-12f172960912 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.362646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.363305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.363583 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.363853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.364732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:16.364910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.366523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.374669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0035164-e64a-4265-9519-a358704a8303 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.378834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.379552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.379847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.380132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.383679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1114/2201] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:16.383679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.392288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cb4b4b-c1a4-4229-ae22-6234d97265c4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.394194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:16.397556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.398254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.398540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.398823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.399578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.401233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.407606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:16.414745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1115/2202] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:16.415690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.429073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8fd980-d156-4898-859b-67aac13170b7 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.435507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.436476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.437113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.437113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.437799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.440190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.472365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1116/2203] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:16 2022] POST /identity/v3/projects => generated 375 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:16.473340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.473715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.475376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.486878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1088/2204] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:16 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:16.487923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.499057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7435e1c7-2583-4d1f-babb-a48bb62218ea service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.568684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fe561d-f2b4-4a95-99ef-83a1a6082839 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.575476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.576976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.577617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.578129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] PATH_INFO: `/v3/projects/cf7747f0d9f8499ab4c82db09a9928e4/users/03fa69d685884c46a0ce1767537467d2/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.609021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf7747f0d9f8499ab4c82db09a9928e4, user_id=03fa69d685884c46a0ce1767537467d2, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.612173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.618903 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.619595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.619880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.620161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.620948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.623447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.641015 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1117/2205] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:16 2022] PUT /identity/v3/projects/cf7747f0d9f8499ab4c82db09a9928e4/users/03fa69d685884c46a0ce1767537467d2/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:16.647141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.680519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1089/2206] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:16 2022] POST /identity/v3/users => generated 288 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:16.682871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.718819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e3ddce-69eb-464d-acb6-2776160200e4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.736759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845ca632-1d61-4a23-b3f5-69c9d77ef517 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.742787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.743540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.744194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.744506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.745367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.747771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.760035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1118/2207] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:16.764821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.792825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6465396e-3077-4d1b-a711-ce7dc9c1910a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.809581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e466d-7c78-490d-879b-6713dad59e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.810527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e466d-7c78-490d-879b-6713dad59e86 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.810821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e466d-7c78-490d-879b-6713dad59e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.811093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e466d-7c78-490d-879b-6713dad59e86 admin admin] PATH_INFO: `/v3/projects/cf7747f0d9f8499ab4c82db09a9928e4/users/03fa69d685884c46a0ce1767537467d2/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.820329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139e466d-7c78-490d-879b-6713dad59e86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf7747f0d9f8499ab4c82db09a9928e4, user_id=03fa69d685884c46a0ce1767537467d2, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.826682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139e466d-7c78-490d-879b-6713dad59e86 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.841909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.842965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.843319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.843711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.844532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.847782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.852493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1119/2208] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:16 2022] PUT /identity/v3/projects/cf7747f0d9f8499ab4c82db09a9928e4/users/03fa69d685884c46a0ce1767537467d2/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:16.858833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1090/2209] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:16.859822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e21e68-f4e4-47c9-b215-3747d03c1842 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.860682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e21e68-f4e4-47c9-b215-3747d03c1842 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.861113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e21e68-f4e4-47c9-b215-3747d03c1842 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.863378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.878985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cb34bf-565f-445b-8ed9-7d590ff943c4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:16.898235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-60e21e68-f4e4-47c9-b215-3747d03c1842 None None] MFA Rules not processed for user `03fa69d685884c46a0ce1767537467d2`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:16.900164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-60e21e68-f4e4-47c9-b215-3747d03c1842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:16.930916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:16.944660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:16.970808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:16.971492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:16.971779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:16.972061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:16.972873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:16.974505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:16.984282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1120/2210] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:16 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:16.990368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1091/2211] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:16.994958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.995234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e21e68-f4e4-47c9-b215-3747d03c1842 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:16.998522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:17.009881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e21e68-f4e4-47c9-b215-3747d03c1842 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.016679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacdceff-e503-423e-8a6e-d0bab87273ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.018315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacdceff-e503-423e-8a6e-d0bab87273ca admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.018607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacdceff-e503-423e-8a6e-d0bab87273ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.018913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacdceff-e503-423e-8a6e-d0bab87273ca admin admin] PATH_INFO: `/v3/projects/42d627e0235a4a4caba84a4057275b5e/users/4d23c3f5f358461da0b1d3ead1121f36/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.037519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacdceff-e503-423e-8a6e-d0bab87273ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=42d627e0235a4a4caba84a4057275b5e, user_id=4d23c3f5f358461da0b1d3ead1121f36, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.040544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacdceff-e503-423e-8a6e-d0bab87273ca admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.060017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1121/2212] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:16 2022] PUT /identity/v3/projects/42d627e0235a4a4caba84a4057275b5e/users/4d23c3f5f358461da0b1d3ead1121f36/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:17.070493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6eec36-2002-446d-ae51-ad5c199e5fd2 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.070820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6eec36-2002-446d-ae51-ad5c199e5fd2 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.071287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6eec36-2002-446d-ae51-ad5c199e5fd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.081268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec546e3-0ec1-40d4-89a1-60fc7b9a54dc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.106901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-df6eec36-2002-446d-ae51-ad5c199e5fd2 None None] MFA Rules not processed for user `4d23c3f5f358461da0b1d3ead1121f36`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:17.108217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-df6eec36-2002-446d-ae51-ad5c199e5fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:17.151923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.153037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.153270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.153551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.219827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1122/2213] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:17 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:17.225429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6eec36-2002-446d-ae51-ad5c199e5fd2 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.240775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.242487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.257059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1092/2214] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:16 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:17.310571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6eec36-2002-446d-ae51-ad5c199e5fd2 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.315510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.316174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.316489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.316773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.317471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.319385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.354578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1123/2215] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:17 2022] POST /identity/v3/projects => generated 375 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:17.360116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.372910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57121ddd-ea07-4ef4-a9f5-955b4a9a983b service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.423243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:17.463259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.463761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.464045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.464324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.466686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.469285 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.493225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1093/2216] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:17 2022] POST /identity/v3/users => generated 288 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:17.497364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.508510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc22694a-bf39-4432-b957-fee731823c61 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.513754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.514615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.515170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.515377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.516023 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.518817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.532697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1124/2217] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:17.534038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.548630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83249a97-cffd-4ef5-9314-8ea8451eb3ff admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.553511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf593ec5-ccb9-459d-ae8e-aa952a15d3a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.554204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf593ec5-ccb9-459d-ae8e-aa952a15d3a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.554465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf593ec5-ccb9-459d-ae8e-aa952a15d3a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.554736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf593ec5-ccb9-459d-ae8e-aa952a15d3a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.555765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf593ec5-ccb9-459d-ae8e-aa952a15d3a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.561710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf593ec5-ccb9-459d-ae8e-aa952a15d3a2 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.576082 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1094/2218] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:17.576932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.587646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf230b-d04c-468b-be1f-4c588cfd9537 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.601804 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.602713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.603000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.603284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] PATH_INFO: `/v3/projects/a3d80ef13e1c4fa7b79d25f5e04a7fa4/users/99a8143822414ba9968f712e776586fa/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.607573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:17.624690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3d80ef13e1c4fa7b79d25f5e04a7fa4, user_id=99a8143822414ba9968f712e776586fa, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.627524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.642812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1125/2219] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:17 2022] PUT /identity/v3/projects/a3d80ef13e1c4fa7b79d25f5e04a7fa4/users/99a8143822414ba9968f712e776586fa/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:17.653547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a46410-7491-4393-9d48-4fa19cc86952 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.653835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a46410-7491-4393-9d48-4fa19cc86952 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.654119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a46410-7491-4393-9d48-4fa19cc86952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.688427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-44a46410-7491-4393-9d48-4fa19cc86952 None None] MFA Rules not processed for user `99a8143822414ba9968f712e776586fa`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:17.689675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-44a46410-7491-4393-9d48-4fa19cc86952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:17.765083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1095/2220] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:17 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:17.769497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.845194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a46410-7491-4393-9d48-4fa19cc86952 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.850070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f5e665-c113-449c-a50d-0e140365e08d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.855504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.856867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.857156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.857435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.857667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a46410-7491-4393-9d48-4fa19cc86952 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.858130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.860344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.867663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.868667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.869362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.869802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] PATH_INFO: `/v3/users/b8354a0625894297b21d10d5031d900f` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.871808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8354a0625894297b21d10d5031d900f)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.874169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.888235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1126/2221] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:17 2022] POST /identity/v3/projects => generated 375 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:17.892166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:17.934515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8354a0625894297b21d10d5031d900f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:17.942836 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8354a0625894297b21d10d5031d900f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:17.943551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8354a0625894297b21d10d5031d900f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:17.944232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8354a0625894297b21d10d5031d900f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:17.962820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c32f439d-bf14-4d7f-a803-9e95d6c96d23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8354a0625894297b21d10d5031d900f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:17.974197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fe3025-6f0b-483a-8360-dda35c558678 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:17.979064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1096/2222] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:17 2022] DELETE /identity/v3/users/b8354a0625894297b21d10d5031d900f => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:17.979266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:17.979836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.980077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.980357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:17.981141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:17.983347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:17.983493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:17.983640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:17.983941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.002380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:18.003612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.013862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:18.020022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1127/2223] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:17 2022] POST /identity/v3/users => generated 294 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:18.021097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.023178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.037413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4272fc-8e48-41f3-bb8a-c2e8016c7e84 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.078912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1097/2224] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:18.089985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.092540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.101265 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.101945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.102667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.102998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.103740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.105386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.106860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:18.120793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1128/2225] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:18 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:18.125520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.140171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29226a7-9d52-4052-b817-b7a0c09f5fd3 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.146284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.147184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.147476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.147761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] PATH_INFO: `/v3/projects/5350d7b1d1f54c8880b1c2f6c2d1c31a/users/ebdfcd8dca69452ca8a85aaabef64b48/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.164245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5350d7b1d1f54c8880b1c2f6c2d1c31a, user_id=ebdfcd8dca69452ca8a85aaabef64b48, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.166986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.168706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beca994c-c07a-4164-b34f-a788b6773b23 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.172728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.173389 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.173674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.173960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.182415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1129/2226] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:18 2022] PUT /identity/v3/projects/5350d7b1d1f54c8880b1c2f6c2d1c31a/users/ebdfcd8dca69452ca8a85aaabef64b48/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:18.186981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.187922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.189444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.203169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1098/2227] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:18.263007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd59a85-cdb1-42e6-837f-52943644561d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.268482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98230ab-acc5-432d-afcf-6caff77f2a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.269152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98230ab-acc5-432d-afcf-6caff77f2a73 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.269442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98230ab-acc5-432d-afcf-6caff77f2a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.269719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98230ab-acc5-432d-afcf-6caff77f2a73 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.270464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98230ab-acc5-432d-afcf-6caff77f2a73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.272271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98230ab-acc5-432d-afcf-6caff77f2a73 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.284233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1130/2228] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:18 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:18.288238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.302553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79a049a-429d-4977-b4dc-0ddbf6708fe0 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.308528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.310149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.310438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.310718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] PATH_INFO: `/v3/projects/5350d7b1d1f54c8880b1c2f6c2d1c31a/users/ebdfcd8dca69452ca8a85aaabef64b48/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.313851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5350d7b1d1f54c8880b1c2f6c2d1c31a, user_id=ebdfcd8dca69452ca8a85aaabef64b48, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.317264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.343656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.344411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.344714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.346452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1099/2229] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:18 2022] PUT /identity/v3/projects/5350d7b1d1f54c8880b1c2f6c2d1c31a/users/ebdfcd8dca69452ca8a85aaabef64b48/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:18.384108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] MFA Rules not processed for user `ebdfcd8dca69452ca8a85aaabef64b48`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:18.385455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.388302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.390157 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.510169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1131/2230] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:18 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:18.522113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.523509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.525140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc024fc-6aa1-40e3-a54f-d4bd9881a54d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.531272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.532730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.533110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.533653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] PATH_INFO: `/v3/users/377f96c18d264057aec1dd7ea4217072` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.535194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=377f96c18d264057aec1dd7ea4217072)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.539948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.587287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c31157-b7d3-4f41-84ba-b84f625f7629 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.592343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9269aed0-6358-4799-bab0-e1b8b71a074c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.592904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9269aed0-6358-4799-bab0-e1b8b71a074c service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.593175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9269aed0-6358-4799-bab0-e1b8b71a074c service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.593728 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9269aed0-6358-4799-bab0-e1b8b71a074c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.595745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9269aed0-6358-4799-bab0-e1b8b71a074c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.610069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:18.623346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '377f96c18d264057aec1dd7ea4217072'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.625585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:18.633803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '377f96c18d264057aec1dd7ea4217072'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.635326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '377f96c18d264057aec1dd7ea4217072'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.636187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '377f96c18d264057aec1dd7ea4217072'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.657397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '377f96c18d264057aec1dd7ea4217072'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.675187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1100/2231] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:18 2022] DELETE /identity/v3/users/377f96c18d264057aec1dd7ea4217072 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:18.675825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.677203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.689052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9269aed0-6358-4799-bab0-e1b8b71a074c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.692250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9269aed0-6358-4799-bab0-e1b8b71a074c service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.705114 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1132/2232] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:18 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:18.706339 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.706702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.707056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.731040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:18.733169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.757344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f4c05-acc1-41da-a466-401d8564e6b6 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:18.763584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:18.764746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:18.765063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:18.765524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] PATH_INFO: `/v3/projects/c361030bed824e3290f7769e28c6f4af` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:18.767069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c361030bed824e3290f7769e28c6f4af)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:18.769227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:18.836549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1133/2233] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:18.843774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c361030bed824e3290f7769e28c6f4af', 'request_id': 'req-4bc4b827-7d3c-4df2-861a-99189358ac60'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.852493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c361030bed824e3290f7769e28c6f4af', 'request_id': 'req-4bc4b827-7d3c-4df2-861a-99189358ac60'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.857162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.859246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.860953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c361030bed824e3290f7769e28c6f4af', 'request_id': 'req-4bc4b827-7d3c-4df2-861a-99189358ac60'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.916225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] The token cache is being invalidate because project c361030bed824e3290f7769e28c6f4af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:18.926158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.927097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:18.931612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1101/2234] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:18 2022] DELETE /identity/v3/projects/c361030bed824e3290f7769e28c6f4af => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:18.933625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:18.936361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:18.946897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146633b3-5c11-47e4-a32e-eaaf9e84d65e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:19.014872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:19.027671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc4b827-7d3c-4df2-861a-99189358ac60 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:19.030794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:19.031509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.031976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.032326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.034174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:19.034313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:19.035271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.035654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.036027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] PATH_INFO: `/v3/users/03fa69d685884c46a0ce1767537467d2` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.037438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03fa69d685884c46a0ce1767537467d2)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:19.039551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:19.050447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:19.051904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:19.063258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1134/2235] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:19.128798 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03fa69d685884c46a0ce1767537467d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.135081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03fa69d685884c46a0ce1767537467d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.135937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03fa69d685884c46a0ce1767537467d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.142069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03fa69d685884c46a0ce1767537467d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.162535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03fa69d685884c46a0ce1767537467d2'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.181497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1102/2236] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:18 2022] DELETE /identity/v3/users/03fa69d685884c46a0ce1767537467d2 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:19.458940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:19.461136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:19.475370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe631f4-1397-4f12-b6e2-6557a1d01951 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:19.490690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:19.561574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:19.563333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:19.574825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:19.575771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.576166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.576623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] PATH_INFO: `/v3/projects/a6f1f3d7ffff4273a18637cf0fb73cbc` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.577830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a6f1f3d7ffff4273a18637cf0fb73cbc)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:19.580497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:19.593985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fe48c1-5620-4184-9e03-1de62fdaa809 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:19.600325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:19.601537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.601824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.602103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] PATH_INFO: `/v3/projects/cf7747f0d9f8499ab4c82db09a9928e4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.603360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf7747f0d9f8499ab4c82db09a9928e4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:19.605349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:19.641653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a6f1f3d7ffff4273a18637cf0fb73cbc', 'request_id': 'req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.648039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a6f1f3d7ffff4273a18637cf0fb73cbc', 'request_id': 'req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.655378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a6f1f3d7ffff4273a18637cf0fb73cbc', 'request_id': 'req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.666939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf7747f0d9f8499ab4c82db09a9928e4', 'request_id': 'req-624a916a-0fd1-4922-af01-e4211e4d3c64'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.669378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] The token cache is being invalidate because project a6f1f3d7ffff4273a18637cf0fb73cbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:19.669672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.670330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.672125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1135/2237] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:19 2022] DELETE /identity/v3/projects/a6f1f3d7ffff4273a18637cf0fb73cbc => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:19.674494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf7747f0d9f8499ab4c82db09a9928e4', 'request_id': 'req-624a916a-0fd1-4922-af01-e4211e4d3c64'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.675754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:19.677143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:19.681869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf7747f0d9f8499ab4c82db09a9928e4', 'request_id': 'req-624a916a-0fd1-4922-af01-e4211e4d3c64'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.700972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] The token cache is being invalidate because project cf7747f0d9f8499ab4c82db09a9928e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:19.701270 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.701919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-624a916a-0fd1-4922-af01-e4211e4d3c64 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.707410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1103/2238] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:19 2022] DELETE /identity/v3/projects/cf7747f0d9f8499ab4c82db09a9928e4 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:19.708790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df14a2-f038-4b22-9b45-f6413a8548b2 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.709083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df14a2-f038-4b22-9b45-f6413a8548b2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.709444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df14a2-f038-4b22-9b45-f6413a8548b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.729444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d5df14a2-f038-4b22-9b45-f6413a8548b2 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:19.730723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d5df14a2-f038-4b22-9b45-f6413a8548b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:19.752287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ee1a2c-c5d3-4718-bf6c-6ed11f979842 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:19.779455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:19.792837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1104/2239] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:19.795616 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:19.802058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a37ef5-0ccd-437f-8424-c1bc5865e421 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.802352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a37ef5-0ccd-437f-8424-c1bc5865e421 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.802651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a37ef5-0ccd-437f-8424-c1bc5865e421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.811954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:19.812925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.813264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.813546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] PATH_INFO: `/v3/users/e6e5d04710784b50b72dabd85b50e123` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.814758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6e5d04710784b50b72dabd85b50e123)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:19.816771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:19.821527 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-24a37ef5-0ccd-437f-8424-c1bc5865e421 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:19.822732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-24a37ef5-0ccd-437f-8424-c1bc5865e421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:19.831058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1105/2240] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:19.835378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a37ef5-0ccd-437f-8424-c1bc5865e421 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:19.845577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a37ef5-0ccd-437f-8424-c1bc5865e421 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:19.850539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:19.851192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.851480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.851760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.852529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:19.852839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:19.854442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:19.865412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6e5d04710784b50b72dabd85b50e123'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.866083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6e5d04710784b50b72dabd85b50e123'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.866794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1106/2241] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:19.870969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:19.872648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:19.875582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6e5d04710784b50b72dabd85b50e123'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.876241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6e5d04710784b50b72dabd85b50e123'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.883836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6e5d04710784b50b72dabd85b50e123'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:19.884969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7d339d-aedc-4253-8eb6-e6cb87f27e95 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:19.889967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:19.890628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:19.890913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:19.891193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:19.891872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:19.893781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:19.901811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1136/2242] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:19 2022] DELETE /identity/v3/users/e6e5d04710784b50b72dabd85b50e123 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:19.924468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1107/2243] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:19 2022] POST /identity/v3/projects => generated 357 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:19.925671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:19.995696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e04f86d-10f4-45cd-be31-730288a9b8dd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.054461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.055203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.055472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.055753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.056472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.059655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.075784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:20.084780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1137/2244] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:19 2022] POST /identity/v3/users => generated 279 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:20.087141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.098704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a63ee0-5935-43d0-a2da-0c0e449d90fb admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.103707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.104406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.104699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.105178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.105710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.107910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.121176 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1108/2245] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:20.122441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.132700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6172a71-6c9d-4dde-8aca-d61bdc68f112 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.137953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.138613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.138894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.139170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.139897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.141525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.151661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1138/2246] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:20.156355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.166576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd23f0e0-3b1b-4ae5-ab48-4a4501d07bdc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.171824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952e1420-07e2-41fe-a4e4-87d5728ddd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.172756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e1420-07e2-41fe-a4e4-87d5728ddd5f admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.173301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e1420-07e2-41fe-a4e4-87d5728ddd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.173626 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952e1420-07e2-41fe-a4e4-87d5728ddd5f admin admin] PATH_INFO: `/v3/projects/f5364f47cbb14476951d037d310b4acc/users/1407a3376126403199648f6a07b951b5/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.190096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952e1420-07e2-41fe-a4e4-87d5728ddd5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5364f47cbb14476951d037d310b4acc, user_id=1407a3376126403199648f6a07b951b5, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.193033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952e1420-07e2-41fe-a4e4-87d5728ddd5f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.197650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.199699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.208735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1109/2247] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:20 2022] PUT /identity/v3/projects/f5364f47cbb14476951d037d310b4acc/users/1407a3376126403199648f6a07b951b5/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:20.215051 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84032691-d5a2-448f-a729-b18e87064dfa admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.220878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.222202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.222849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.253081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] MFA Rules not processed for user `1407a3376126403199648f6a07b951b5`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:20.254301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.293894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.294793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.295098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.295375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] PATH_INFO: `/v3/projects/47b8c0d424dd4f4abeef3fa4b2c327ed` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.296590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47b8c0d424dd4f4abeef3fa4b2c327ed)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.298533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.328505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1110/2248] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:20 2022] POST /identity/v3/auth/tokens => generated 3224 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:20.346968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47b8c0d424dd4f4abeef3fa4b2c327ed', 'request_id': 'req-bac48d58-a96e-487e-8e8c-4f7604ec2be3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.352919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47b8c0d424dd4f4abeef3fa4b2c327ed', 'request_id': 'req-bac48d58-a96e-487e-8e8c-4f7604ec2be3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.359927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.360265 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47b8c0d424dd4f4abeef3fa4b2c327ed', 'request_id': 'req-bac48d58-a96e-487e-8e8c-4f7604ec2be3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.361353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.371883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] The token cache is being invalidate because project 47b8c0d424dd4f4abeef3fa4b2c327ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:20.372168 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.372852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bac48d58-a96e-487e-8e8c-4f7604ec2be3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.374622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1139/2249] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:20 2022] DELETE /identity/v3/projects/47b8c0d424dd4f4abeef3fa4b2c327ed => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:20.379126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70231a27-d4bf-4950-9394-51b8337de182 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.379493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70231a27-d4bf-4950-9394-51b8337de182 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.379782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70231a27-d4bf-4950-9394-51b8337de182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.397870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-70231a27-d4bf-4950-9394-51b8337de182 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:20.399052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-70231a27-d4bf-4950-9394-51b8337de182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.419044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee6c3a5-ea9e-4696-8a32-f38d4a251f20 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.439349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:20.461560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1140/2250] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:20.470298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2e7cd-14f8-4e06-a802-28ab2b2fa396 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.470620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2e7cd-14f8-4e06-a802-28ab2b2fa396 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.470900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b2e7cd-14f8-4e06-a802-28ab2b2fa396 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.473953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.474604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.474918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.475192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.476970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.488864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b7b2e7cd-14f8-4e06-a802-28ab2b2fa396 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:20.490044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b7b2e7cd-14f8-4e06-a802-28ab2b2fa396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.498479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1141/2251] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:20.502426 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b2e7cd-14f8-4e06-a802-28ab2b2fa396 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.511467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b2e7cd-14f8-4e06-a802-28ab2b2fa396 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.516916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.517552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.517902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.518192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.518951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:20.519264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.520915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.530075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:20.540020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1142/2252] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:20.541096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.544937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.546404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.553371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a924bf7a-ae37-4877-8341-c6e4a0cc18ed admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.558148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1111/2253] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:20 2022] GET /identity/v3/auth/tokens => generated 3224 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:20.558391 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.559057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.559886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.560248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.560986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.562902 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.575731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.577946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.595448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15726055-c244-4775-9c0a-49b7a4363f71 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.605706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1143/2254] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:20 2022] POST /identity/v3/projects => generated 387 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:20.605706 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.605899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.605899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.605899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.605899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] PATH_INFO: `/v3/users/1407a3376126403199648f6a07b951b5` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.606929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] RBAC: Authorizing `identity:delete_user(user_id=1407a3376126403199648f6a07b951b5)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.609855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.670283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1407a3376126403199648f6a07b951b5'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.672900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a82384-05c0-4539-8102-7ded172ac9e7 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.678183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.678811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.679089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.679366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.680330 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.682144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.682269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1407a3376126403199648f6a07b951b5'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.682991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1407a3376126403199648f6a07b951b5'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.683697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1407a3376126403199648f6a07b951b5'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.701637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c1f28f8e-c13b-4fd2-a687-823107f928be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1407a3376126403199648f6a07b951b5'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:20.710119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1144/2255] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:20 2022] POST /identity/v3/users => generated 294 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:20.714274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.716220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.721820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1112/2256] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:20 2022] DELETE /identity/v3/users/1407a3376126403199648f6a07b951b5 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:20.726162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.727058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.727345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.729192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc94834-0fa6-44c3-b0bf-0257d38f8e7e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.748974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:20.750765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.796786 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.797477 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.797765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.798049 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.798825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.802899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.813758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1145/2257] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:20.815506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:20.818818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.835351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c321d18-a3d4-481d-b395-22aebc5de8a2 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.840609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1113/2258] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:20.844079 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.845004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.845444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.845876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.846960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.849448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.858668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.860121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.863358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1146/2259] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:20.868667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:20.882588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6313ad5-4cae-409a-90eb-6804ec3bc857 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.890452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a70aa2-2a61-429e-a1a2-2dc0291bebd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.891537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a70aa2-2a61-429e-a1a2-2dc0291bebd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.891952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a70aa2-2a61-429e-a1a2-2dc0291bebd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.892457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a70aa2-2a61-429e-a1a2-2dc0291bebd8 admin admin] PATH_INFO: `/v3/projects/a72bf0961877483b85f836ca4d950f12/users/9d869c9e9fe74a1cbff8da4e94540109/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.912162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a70aa2-2a61-429e-a1a2-2dc0291bebd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a72bf0961877483b85f836ca4d950f12, user_id=9d869c9e9fe74a1cbff8da4e94540109, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:20.915233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a70aa2-2a61-429e-a1a2-2dc0291bebd8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:20.949825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd619b42-3f74-4bff-9276-43bfbfd75974 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:20.950157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1147/2260] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:20 2022] PUT /identity/v3/projects/a72bf0961877483b85f836ca4d950f12/users/9d869c9e9fe74a1cbff8da4e94540109/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:20.959899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b2e31-74b6-489d-b5aa-e53f67609db7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.960281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b2e31-74b6-489d-b5aa-e53f67609db7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.960673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43b2e31-74b6-489d-b5aa-e53f67609db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:20.989100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a43b2e31-74b6-489d-b5aa-e53f67609db7 None None] MFA Rules not processed for user `9d869c9e9fe74a1cbff8da4e94540109`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:20.990337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a43b2e31-74b6-489d-b5aa-e53f67609db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:20.998474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:20.999142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:20.999434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:20.999759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.010739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:21.067724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.069195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.073506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1148/2261] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:21.078280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43b2e31-74b6-489d-b5aa-e53f67609db7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.083580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1114/2262] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:21.091474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43b2e31-74b6-489d-b5aa-e53f67609db7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.096200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.096890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.097234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.097433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.098119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.102501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.129782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1149/2263] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:21 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:21.133742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.144524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d281a879-84cc-494b-9bae-7f90b69ae496 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.235548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:21.249138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.254177 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.254481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.256050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.257131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.260214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.311180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1115/2264] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:21 2022] POST /identity/v3/users => generated 299 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:21.315725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.333632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab97a344-7ab0-4119-9b91-deec2e6be99e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.347913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.348709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.348997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.349279 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.350041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.352310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.368166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1150/2265] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:21.372669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.384680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886bc771-bfa5-41e0-bdd0-56345bff9f81 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.395114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.396511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.396971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.397395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] PATH_INFO: `/v3/projects/6bdb968221044424aa14be24c03981a8/users/cc50610a27da462eaa2f69bd0e1ea25c/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.411179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.413259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:21.417835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bdb968221044424aa14be24c03981a8, user_id=cc50610a27da462eaa2f69bd0e1ea25c, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.423111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.444539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64956f42-1a45-43f3-ba44-85ab3f10cf06 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.445735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1116/2266] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:21 2022] PUT /identity/v3/projects/6bdb968221044424aa14be24c03981a8/users/cc50610a27da462eaa2f69bd0e1ea25c/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:21.451736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.543073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.544783 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.545197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.545571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] PATH_INFO: `/v3/projects/f5364f47cbb14476951d037d310b4acc` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.546883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5364f47cbb14476951d037d310b4acc)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.549248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.557077 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70a4d05-b4c8-414a-8577-8322b8449413 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.563509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.564319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.564781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.565202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.566212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.570043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.601732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1117/2267] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:21.602673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.620772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ea13a5-b10c-4fc1-bbd1-4dbb586a988b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.631900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.632869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.633163 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.633439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] PATH_INFO: `/v3/projects/6bdb968221044424aa14be24c03981a8/users/cc50610a27da462eaa2f69bd0e1ea25c/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.635741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bdb968221044424aa14be24c03981a8, user_id=cc50610a27da462eaa2f69bd0e1ea25c, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.639840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.652684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5364f47cbb14476951d037d310b4acc', 'request_id': 'req-a83e9777-bfee-44c5-b7df-c170a0d304f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:21.652934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1118/2268] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:21 2022] PUT /identity/v3/projects/6bdb968221044424aa14be24c03981a8/users/cc50610a27da462eaa2f69bd0e1ea25c/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:21.654290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.656582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:21.660746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5364f47cbb14476951d037d310b4acc', 'request_id': 'req-a83e9777-bfee-44c5-b7df-c170a0d304f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:21.676683 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5364f47cbb14476951d037d310b4acc', 'request_id': 'req-a83e9777-bfee-44c5-b7df-c170a0d304f5'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:21.706001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] The token cache is being invalidate because project f5364f47cbb14476951d037d310b4acc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:21.706745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:21.708299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a83e9777-bfee-44c5-b7df-c170a0d304f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:21.710109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1151/2269] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:21 2022] DELETE /identity/v3/projects/f5364f47cbb14476951d037d310b4acc => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:21.711571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417980d2-60c1-4aad-bc1a-8557c0e12080 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.711853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417980d2-60c1-4aad-bc1a-8557c0e12080 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.712129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417980d2-60c1-4aad-bc1a-8557c0e12080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.742958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4e466d-c24d-49c4-b1a8-ea32b6d424d4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.749534 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.751515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.751818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.752089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.755193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:21.756989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-417980d2-60c1-4aad-bc1a-8557c0e12080 None None] MFA Rules not processed for user `cc50610a27da462eaa2f69bd0e1ea25c`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:21.758616 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-417980d2-60c1-4aad-bc1a-8557c0e12080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:21.764591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:21.857950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.859342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.865234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1152/2270] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:21 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:21.866661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.867059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.867901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.873780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:21.881946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1119/2271] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:21 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:21.883551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:21.888371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:21.888965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:21.889797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:21.907750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76810aa-86a4-48dd-ac1e-21bc3c762dd2 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:21.919451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1940a990-0dd1-4d3b-b7ec-9cf8aa843e01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:21.920381 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1940a990-0dd1-4d3b-b7ec-9cf8aa843e01 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:21.920875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1940a990-0dd1-4d3b-b7ec-9cf8aa843e01 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:21.921777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1940a990-0dd1-4d3b-b7ec-9cf8aa843e01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:21.933986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1940a990-0dd1-4d3b-b7ec-9cf8aa843e01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:21.935696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1940a990-0dd1-4d3b-b7ec-9cf8aa843e01 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:21.946644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1120/2272] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:21 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:21.994634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1153/2273] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:22.006085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4001aea6-bc4b-4088-850e-b37c7312eeac None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.006397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4001aea6-bc4b-4088-850e-b37c7312eeac None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.006681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4001aea6-bc4b-4088-850e-b37c7312eeac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.025076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.027620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.035858 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-4001aea6-bc4b-4088-850e-b37c7312eeac None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:22.037317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4001aea6-bc4b-4088-850e-b37c7312eeac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.048953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1121/2274] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:22.052622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aba43e-8993-4b1a-b5bc-2f9870d7b525 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.052785 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4001aea6-bc4b-4088-850e-b37c7312eeac None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.057880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.058777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.060491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.060491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] PATH_INFO: `/v3/users/9d869c9e9fe74a1cbff8da4e94540109` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.061415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d869c9e9fe74a1cbff8da4e94540109)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.063371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.067431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4001aea6-bc4b-4088-850e-b37c7312eeac None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.073249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.074182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.074182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.074457 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.075792 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:22.076099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.077772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.091144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1122/2275] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:22.095656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.108926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba7813-7504-40c3-8120-45249d2dd4d2 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.119592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.120006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.120524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.121589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.123481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.126050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.145558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d869c9e9fe74a1cbff8da4e94540109'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:22.146275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d869c9e9fe74a1cbff8da4e94540109'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:22.158198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d869c9e9fe74a1cbff8da4e94540109'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:22.159072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d869c9e9fe74a1cbff8da4e94540109'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:22.173891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-dce107a0-697f-4dff-b1fa-4c1478db6ee8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d869c9e9fe74a1cbff8da4e94540109'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:22.178761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1123/2276] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:22 2022] POST /identity/v3/projects => generated 373 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:22.186273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.194806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.205112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1154/2277] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:22 2022] DELETE /identity/v3/users/9d869c9e9fe74a1cbff8da4e94540109 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:22.209717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.210014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.210651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.232917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:22.234466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.247670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:22.304657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea72525f-9fe7-435d-a0c8-6031fef07be4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.330508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1155/2278] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:22.337568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.340207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.411160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.411905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.412255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.412575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.413329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.415320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.443871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1124/2279] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:22 2022] POST /identity/v3/users => generated 287 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:22.445023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.458765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c26938-a4e7-46be-babb-c58837c0ba9a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.463141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2657c47-5f21-4ed5-97c8-e125d21d4478 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.464509 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.464885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.465183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.465499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.466375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.468424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.475785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.476839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.477128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.477413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.482997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1125/2280] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:22.485508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.493298 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.495121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.498513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa79bdf-c47e-4eee-98a8-48a94b48cc7f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.503982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352e1f41-bdd6-481e-907d-4b650c1b66cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.504183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e1f41-bdd6-481e-907d-4b650c1b66cd admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.504789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e1f41-bdd6-481e-907d-4b650c1b66cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.505152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352e1f41-bdd6-481e-907d-4b650c1b66cd admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.505951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352e1f41-bdd6-481e-907d-4b650c1b66cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.506641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1156/2281] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:22.507759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352e1f41-bdd6-481e-907d-4b650c1b66cd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.520437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1126/2282] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:22 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:22.524546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.547842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6529641b-896f-49ec-9b3a-fc029bb2b408 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.554618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.555690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.555973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.556255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] PATH_INFO: `/v3/projects/81cf59b04ff94e439d5d3dc261d5b44e/users/80de1d15fb47407993d515c3d4ccd3b9/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.574415 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=81cf59b04ff94e439d5d3dc261d5b44e, user_id=80de1d15fb47407993d515c3d4ccd3b9, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.577821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.596175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1157/2283] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:22 2022] PUT /identity/v3/projects/81cf59b04ff94e439d5d3dc261d5b44e/users/80de1d15fb47407993d515c3d4ccd3b9/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:22.607567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.607996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.608317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.645647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] MFA Rules not processed for user `80de1d15fb47407993d515c3d4ccd3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:22.647472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.661240 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:22.765417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1127/2284] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:22 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:22.769821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.831757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.833216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.881968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331066e0-e676-49da-a5bd-3b2cdecccb5d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.888750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.889425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.889856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.890162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.891040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:22.893139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:22.920441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f80b11-64ac-4b77-b7df-ea63c7fada90 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:22.926003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1158/2285] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:22 2022] POST /identity/v3/projects => generated 371 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:22.926861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:22.933669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.935796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:22.936751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:22.937038 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:22.937320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:22.939069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:22.958486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:23.009187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.010731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.010950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ce4f70-f1fe-48c6-8277-915884bfdb2f admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.015676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.016575 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.016861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.017140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] PATH_INFO: `/v3/projects/a72bf0961877483b85f836ca4d950f12` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.018324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a72bf0961877483b85f836ca4d950f12)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.020226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.021479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1128/2286] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:22 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:23.022387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.033387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d5f97-32ce-43bb-b805-8dfea5e5b614 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.038047 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.038710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.039005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.039295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.039970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.041926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.064432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1129/2287] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:23 2022] POST /identity/v3/users => generated 292 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:23.068535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.079993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a18c1b1-7d6b-45e7-b0d3-92a05ee0ef4f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.081237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a72bf0961877483b85f836ca4d950f12', 'request_id': 'req-1ff5cdf9-7688-4887-8fa9-834c98b0b701'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.092251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a72bf0961877483b85f836ca4d950f12', 'request_id': 'req-1ff5cdf9-7688-4887-8fa9-834c98b0b701'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.102215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a72bf0961877483b85f836ca4d950f12', 'request_id': 'req-1ff5cdf9-7688-4887-8fa9-834c98b0b701'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.116101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] The token cache is being invalidate because project a72bf0961877483b85f836ca4d950f12 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:23.116430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.117213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.118941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1159/2288] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:22 2022] DELETE /identity/v3/projects/a72bf0961877483b85f836ca4d950f12 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:23.131569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.131569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.133044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:23.153691 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.154574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.154868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.155153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.155893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.157594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.170816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1130/2289] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:23 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:23.171881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.173720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.186725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece8782-ecf2-43d3-879f-4ee41a6552ad admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.191558 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.192469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.192764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.193235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] PATH_INFO: `/v3/projects/248155f307ff41c4bf5be750535f8991/users/ac15d90ddb5d4cb8ba2a8523e999ccb9/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.196158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff5cdf9-7688-4887-8fa9-834c98b0b701 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.202171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.203082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.203365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.203796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] PATH_INFO: `/v3/users/cc50610a27da462eaa2f69bd0e1ea25c` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.205315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc50610a27da462eaa2f69bd0e1ea25c)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.207297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.211982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=248155f307ff41c4bf5be750535f8991, user_id=ac15d90ddb5d4cb8ba2a8523e999ccb9, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.215136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.231025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1131/2290] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:23 2022] PUT /identity/v3/projects/248155f307ff41c4bf5be750535f8991/users/ac15d90ddb5d4cb8ba2a8523e999ccb9/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:23.235160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.264806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc50610a27da462eaa2f69bd0e1ea25c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.265757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc50610a27da462eaa2f69bd0e1ea25c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.268830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:23.279318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc50610a27da462eaa2f69bd0e1ea25c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.280931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc50610a27da462eaa2f69bd0e1ea25c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.290169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc50610a27da462eaa2f69bd0e1ea25c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.306610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c30440-92be-4b8b-b542-8350d60c49a6 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.312216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd71fcb-c962-460a-b29e-c04a36fee3fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.312941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd71fcb-c962-460a-b29e-c04a36fee3fe admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.313282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd71fcb-c962-460a-b29e-c04a36fee3fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.316210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd71fcb-c962-460a-b29e-c04a36fee3fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.316210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd71fcb-c962-460a-b29e-c04a36fee3fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.316210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd71fcb-c962-460a-b29e-c04a36fee3fe admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.317266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1160/2291] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:23 2022] DELETE /identity/v3/users/cc50610a27da462eaa2f69bd0e1ea25c => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:23.332928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1132/2292] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:23 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:23.335348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.337057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.431934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91f4a03-9ffe-40dc-9c8e-0240799d3e24 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.437245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.438219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.438553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.438835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] PATH_INFO: `/v3/projects/248155f307ff41c4bf5be750535f8991/users/ac15d90ddb5d4cb8ba2a8523e999ccb9/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.441170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] RBAC: Authorizing `identity:create_grant(project_id=248155f307ff41c4bf5be750535f8991, user_id=ac15d90ddb5d4cb8ba2a8523e999ccb9, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.443650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.468473 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1161/2293] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:23 2022] PUT /identity/v3/projects/248155f307ff41c4bf5be750535f8991/users/ac15d90ddb5d4cb8ba2a8523e999ccb9/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:23.538146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.538929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.539379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.570886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] MFA Rules not processed for user `ac15d90ddb5d4cb8ba2a8523e999ccb9`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:23.573791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.620261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.623371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.676044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1133/2294] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:23 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:23.688818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:23.690377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.711102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74268020-190c-4372-830d-e6dab76dabda admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.715898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.716888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.717333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.718445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] PATH_INFO: `/v3/projects/6bdb968221044424aa14be24c03981a8` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.720269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bdb968221044424aa14be24c03981a8)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.722861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.754879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:23.780129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aff8d06-5473-4010-a121-31eb6fc9657e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:23.792696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bdb968221044424aa14be24c03981a8', 'request_id': 'req-ea3d8668-a41d-4bf8-9e9c-171743f8b790'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.799941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bdb968221044424aa14be24c03981a8', 'request_id': 'req-ea3d8668-a41d-4bf8-9e9c-171743f8b790'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.805191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:23.809897 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bdb968221044424aa14be24c03981a8', 'request_id': 'req-ea3d8668-a41d-4bf8-9e9c-171743f8b790'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.827749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] The token cache is being invalidate because project 6bdb968221044424aa14be24c03981a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:23.828075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.828864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ea3d8668-a41d-4bf8-9e9c-171743f8b790 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:23.834504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1162/2295] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:23 2022] DELETE /identity/v3/projects/6bdb968221044424aa14be24c03981a8 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:23.842050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba50705-14e5-49fa-8c91-eed4e92734b0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.842357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba50705-14e5-49fa-8c91-eed4e92734b0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.842606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba50705-14e5-49fa-8c91-eed4e92734b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.844337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef9279f-0ed2-4523-b29a-4daa7805233a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:23.845089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9279f-0ed2-4523-b29a-4daa7805233a service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.845390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9279f-0ed2-4523-b29a-4daa7805233a service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.845654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef9279f-0ed2-4523-b29a-4daa7805233a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.848138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9ef9279f-0ed2-4523-b29a-4daa7805233a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.864062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-bba50705-14e5-49fa-8c91-eed4e92734b0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:23.865469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bba50705-14e5-49fa-8c91-eed4e92734b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:23.939904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef9279f-0ed2-4523-b29a-4daa7805233a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:23.941396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef9279f-0ed2-4523-b29a-4daa7805233a service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:23.951719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1134/2296] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:23 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:23.968545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1163/2297] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:23.975742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c238c968-8555-48fc-a665-1e131f4fb364 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:23.975742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c238c968-8555-48fc-a665-1e131f4fb364 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:23.975742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c238c968-8555-48fc-a665-1e131f4fb364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:23.999039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c238c968-8555-48fc-a665-1e131f4fb364 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:24.000284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c238c968-8555-48fc-a665-1e131f4fb364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:24.011146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1135/2298] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:24.015652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba50705-14e5-49fa-8c91-eed4e92734b0 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.034652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba50705-14e5-49fa-8c91-eed4e92734b0 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.040981 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.041778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.042244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.042678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.043524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:24.044601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.046487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.068971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1164/2299] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:24.074011 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c238c968-8555-48fc-a665-1e131f4fb364 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.084630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c238c968-8555-48fc-a665-1e131f4fb364 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.089419 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.090076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.090361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.090680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.091823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.093832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.134001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1136/2300] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:24 2022] POST /identity/v3/projects => generated 387 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:24.144979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.155229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6603fe82-aee3-4e47-abbd-4c8aef514d6b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.230917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.230917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.230917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.232331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.233078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.235027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.262863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1165/2301] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:24 2022] POST /identity/v3/users => generated 294 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:24.267316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.280111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c24127e-f3e8-43a0-a6e0-0f18f7f0fdf1 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.286425 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.287222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.287387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.287745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.288415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.290684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.311838 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1137/2302] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:24 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:24.312825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.328711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b716ae2-826e-4b36-9dcb-f493c5f32c2e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.338606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74633784-08e1-4f4d-9e10-726d94cabaca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.339306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74633784-08e1-4f4d-9e10-726d94cabaca admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.341327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74633784-08e1-4f4d-9e10-726d94cabaca admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.343985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74633784-08e1-4f4d-9e10-726d94cabaca admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.344994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74633784-08e1-4f4d-9e10-726d94cabaca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.349045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74633784-08e1-4f4d-9e10-726d94cabaca admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.373512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1166/2303] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:24 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:24.389501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.400508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8ccd75-2247-4556-8b2a-c81e9affca3b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.419938 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.420882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.421170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.421565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] PATH_INFO: `/v3/projects/9b4955df8ffa4df7a721a79fff26967d/users/b94cd28ed2234dfb814d7a4a3fd96b76/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.446147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b4955df8ffa4df7a721a79fff26967d, user_id=b94cd28ed2234dfb814d7a4a3fd96b76, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.449304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.475275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1138/2304] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:24 2022] PUT /identity/v3/projects/9b4955df8ffa4df7a721a79fff26967d/users/b94cd28ed2234dfb814d7a4a3fd96b76/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:24.477009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de31fda-3033-4e07-919f-86213dfee2c9 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.477292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de31fda-3033-4e07-919f-86213dfee2c9 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.477579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de31fda-3033-4e07-919f-86213dfee2c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.522305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5de31fda-3033-4e07-919f-86213dfee2c9 None None] MFA Rules not processed for user `b94cd28ed2234dfb814d7a4a3fd96b76`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:24.525707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5de31fda-3033-4e07-919f-86213dfee2c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:24.625354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:24.665854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1167/2305] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:24.666603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.722782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:24.743697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe37217f-c443-4e10-889f-c3c7dbcaaadc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.748373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.749055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.749337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.749610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.750298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.752317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.777769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1139/2306] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:24 2022] POST /identity/v3/projects => generated 385 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:24.782373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de31fda-3033-4e07-919f-86213dfee2c9 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.793045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de31fda-3033-4e07-919f-86213dfee2c9 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.849578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.850249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.850533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.850810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.851488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.853453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.871898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1168/2307] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:24 2022] POST /identity/v3/users => generated 299 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:24.875809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.886057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b583e9-6e71-485b-b849-90ce25702aec admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.891230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.891894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.892179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.892498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.893247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.894831 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.906954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1140/2308] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:24.910946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.920791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1114d4d0-9206-44bc-a536-91a9b45d900d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:24.925582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:24.926451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:24.926777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:24.927129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] PATH_INFO: `/v3/projects/58cf2c034be44d7284e30f5099381d20/users/5b9d0eb8134443e7ad8c0608f43db6f7/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:24.943000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58cf2c034be44d7284e30f5099381d20, user_id=5b9d0eb8134443e7ad8c0608f43db6f7, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:24.945701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:24.959181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1169/2309] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:24 2022] PUT /identity/v3/projects/58cf2c034be44d7284e30f5099381d20/users/5b9d0eb8134443e7ad8c0608f43db6f7/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:24.963300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:24.974667 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c582b8-00df-4ec3-8ddc-e9995127380c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:25.031882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a9314c-470a-4a6a-bae9-ee830c46b7c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:25.032699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a9314c-470a-4a6a-bae9-ee830c46b7c0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:25.033004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a9314c-470a-4a6a-bae9-ee830c46b7c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:25.033322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a9314c-470a-4a6a-bae9-ee830c46b7c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:25.034292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a9314c-470a-4a6a-bae9-ee830c46b7c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:25.035643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a9314c-470a-4a6a-bae9-ee830c46b7c0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:25.045969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1141/2310] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:24 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:25.050713 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:25.061666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932923ec-598e-46b1-b422-4b2c584b0301 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:25.066326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:25.067235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:25.067517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:25.067792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] PATH_INFO: `/v3/projects/58cf2c034be44d7284e30f5099381d20/users/5b9d0eb8134443e7ad8c0608f43db6f7/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:25.069832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=58cf2c034be44d7284e30f5099381d20, user_id=5b9d0eb8134443e7ad8c0608f43db6f7, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:25.072527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:25.083554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1170/2311] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:25 2022] PUT /identity/v3/projects/58cf2c034be44d7284e30f5099381d20/users/5b9d0eb8134443e7ad8c0608f43db6f7/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:25.094577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a211ab-6ea9-4803-8a8e-3b363dabab80 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:25.094875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a211ab-6ea9-4803-8a8e-3b363dabab80 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:25.095291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a211ab-6ea9-4803-8a8e-3b363dabab80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:25.120899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-48a211ab-6ea9-4803-8a8e-3b363dabab80 None None] MFA Rules not processed for user `5b9d0eb8134443e7ad8c0608f43db6f7`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:25.122608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-48a211ab-6ea9-4803-8a8e-3b363dabab80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:25.189034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1142/2312] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:25 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:25.199837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:25.202160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:25.259255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1891c43-b319-4bb5-aca9-b00338a069ad admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:25.263663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340a8d53-1bff-443e-b054-d88cc4196c4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:25.264304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a8d53-1bff-443e-b054-d88cc4196c4a service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:25.264619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a8d53-1bff-443e-b054-d88cc4196c4a service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:25.264901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a8d53-1bff-443e-b054-d88cc4196c4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:25.273904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340a8d53-1bff-443e-b054-d88cc4196c4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:25.275346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340a8d53-1bff-443e-b054-d88cc4196c4a service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:25.284947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1171/2313] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:25 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:26.020755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a211ab-6ea9-4803-8a8e-3b363dabab80 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:26.062475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:26.084631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a211ab-6ea9-4803-8a8e-3b363dabab80 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:26.089117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:26.089992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:26.090279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:26.090560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] PATH_INFO: `/v3/users/b94cd28ed2234dfb814d7a4a3fd96b76` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:26.092253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b94cd28ed2234dfb814d7a4a3fd96b76)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:26.094231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:26.141698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b94cd28ed2234dfb814d7a4a3fd96b76'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.148679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b94cd28ed2234dfb814d7a4a3fd96b76'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.149376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b94cd28ed2234dfb814d7a4a3fd96b76'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.149995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b94cd28ed2234dfb814d7a4a3fd96b76'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.163765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b94cd28ed2234dfb814d7a4a3fd96b76'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.177443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1143/2314] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:26 2022] DELETE /identity/v3/users/b94cd28ed2234dfb814d7a4a3fd96b76 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:26.181899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:26.182371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:26.183535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:26.204245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:26.205673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:26.244612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:26.270044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1172/2315] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:26.280353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:26.281791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:26.340156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3aa276-f190-4506-be38-fa1f9b7e2728 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:26.344701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:26.345357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:26.345653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:26.345932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:26.355495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:26.356981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:26.366359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1144/2316] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:26.636479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:26.638659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:26.652034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63370f2a-96a9-456a-9d4e-fc7b6bb8da9c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:26.657416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:26.658284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:26.658563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:26.658834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] PATH_INFO: `/v3/projects/9b4955df8ffa4df7a721a79fff26967d` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:26.660034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b4955df8ffa4df7a721a79fff26967d)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:26.662092 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:26.669192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:26.670676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:26.683506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9a425-29c1-4c69-9418-aaa49cac90c9 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:26.690873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:26.691776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:26.692061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:26.692343 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] PATH_INFO: `/v3/users/80de1d15fb47407993d515c3d4ccd3b9` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:26.693583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=80de1d15fb47407993d515c3d4ccd3b9)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:26.695646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:26.715427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b4955df8ffa4df7a721a79fff26967d', 'request_id': 'req-0738ec24-3d94-45f4-8dec-116351cc0436'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.721509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b4955df8ffa4df7a721a79fff26967d', 'request_id': 'req-0738ec24-3d94-45f4-8dec-116351cc0436'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.731829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b4955df8ffa4df7a721a79fff26967d', 'request_id': 'req-0738ec24-3d94-45f4-8dec-116351cc0436'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.745821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] The token cache is being invalidate because project 9b4955df8ffa4df7a721a79fff26967d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:26.746378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.747570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.749507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1173/2317] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:26 2022] DELETE /identity/v3/projects/9b4955df8ffa4df7a721a79fff26967d => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:26.752984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80de1d15fb47407993d515c3d4ccd3b9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.753130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:26.754373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:26.758338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:26.761183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80de1d15fb47407993d515c3d4ccd3b9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.762051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80de1d15fb47407993d515c3d4ccd3b9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.763946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80de1d15fb47407993d515c3d4ccd3b9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.780155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80de1d15fb47407993d515c3d4ccd3b9'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.803268 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1145/2318] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:26 2022] DELETE /identity/v3/users/80de1d15fb47407993d515c3d4ccd3b9 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:26.804205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:26.805566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:26.832571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0738ec24-3d94-45f4-8dec-116351cc0436 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:26.900714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78756a71-7562-4f95-ad9c-5fbcfe867b5f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:26.905539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:26.906417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:26.906706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:26.906981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] PATH_INFO: `/v3/users/5b9d0eb8134443e7ad8c0608f43db6f7` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:26.908711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b9d0eb8134443e7ad8c0608f43db6f7)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:26.910647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:26.923539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:26.925994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:26.927744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:26.928209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] PATH_INFO: `/v3/users/302c737b117842fa9faed06f0f8331d0` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:26.933208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] RBAC: Authorizing `identity:delete_user(user_id=302c737b117842fa9faed06f0f8331d0)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:26.936282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:26.957406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:26.980468 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b9d0eb8134443e7ad8c0608f43db6f7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.991601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b9d0eb8134443e7ad8c0608f43db6f7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.992633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b9d0eb8134443e7ad8c0608f43db6f7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:26.994478 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b9d0eb8134443e7ad8c0608f43db6f7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.028805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-21954591-b080-4377-ba57-9b5c15e84eec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b9d0eb8134443e7ad8c0608f43db6f7'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.035346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '302c737b117842fa9faed06f0f8331d0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.036342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '302c737b117842fa9faed06f0f8331d0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.047219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1146/2319] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:26 2022] DELETE /identity/v3/users/5b9d0eb8134443e7ad8c0608f43db6f7 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:27.049539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:27.049539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:27.049539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:27.056855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '302c737b117842fa9faed06f0f8331d0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.058351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '302c737b117842fa9faed06f0f8331d0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.077607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-ad5a5b82-264b-416c-9824-9f4828766f66 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '302c737b117842fa9faed06f0f8331d0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.097019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:27.098474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:27.112256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1174/2320] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:26 2022] DELETE /identity/v3/users/302c737b117842fa9faed06f0f8331d0 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:27.116530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:27.117037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:27.117468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:27.152498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:27.154814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:27.178682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:27.201932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1147/2321] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:27.240198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:27.241995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:27.285266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1175/2322] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:27.306021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:27.308535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:27.371450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6354fd8-7c5b-472b-ace1-a87182ad8703 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:27.421003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd99b1e-1445-4893-a782-dde43fd2e3a6 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:27.424930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:27.425777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:27.426561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:27.426561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:27.438024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:27.439669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:27.455150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1176/2323] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:27.475625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11d8a9c-48b3-467b-8da1-dad98dd9e9a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:27.476271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11d8a9c-48b3-467b-8da1-dad98dd9e9a1 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:27.476597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11d8a9c-48b3-467b-8da1-dad98dd9e9a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:27.476878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11d8a9c-48b3-467b-8da1-dad98dd9e9a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:27.487111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11d8a9c-48b3-467b-8da1-dad98dd9e9a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:27.489284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11d8a9c-48b3-467b-8da1-dad98dd9e9a1 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:27.503449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1148/2324] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 264 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:27.704533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:27.706839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:27.722261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6331e08f-f4ea-43c6-a048-3e6655dcff4b service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:27.727157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:27.728106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:27.728414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:27.728765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] PATH_INFO: `/v3/projects/58cf2c034be44d7284e30f5099381d20` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:27.730120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58cf2c034be44d7284e30f5099381d20)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:27.732463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:27.786042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:27.808340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58cf2c034be44d7284e30f5099381d20', 'request_id': 'req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.840267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58cf2c034be44d7284e30f5099381d20', 'request_id': 'req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.854960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58cf2c034be44d7284e30f5099381d20', 'request_id': 'req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.885814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] The token cache is being invalidate because project 58cf2c034be44d7284e30f5099381d20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:27.886120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.888461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:27.898414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1177/2325] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:27 2022] DELETE /identity/v3/projects/58cf2c034be44d7284e30f5099381d20 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:27.900884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be8d485-7819-453e-b63f-f82747023634 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:27.901198 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be8d485-7819-453e-b63f-f82747023634 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:27.901470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be8d485-7819-453e-b63f-f82747023634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:27.945282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5be8d485-7819-453e-b63f-f82747023634 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:27.947861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5be8d485-7819-453e-b63f-f82747023634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:27.985977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:27.991341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.041127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1149/2326] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:28.042077 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be8d485-7819-453e-b63f-f82747023634 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.043576 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5be8d485-7819-453e-b63f-f82747023634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.057042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be8d485-7819-453e-b63f-f82747023634 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.066007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.067093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.067375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.067791 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] PATH_INFO: `/v3/projects/81cf59b04ff94e439d5d3dc261d5b44e` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.069046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] RBAC: Authorizing `identity:delete_project(project_id=81cf59b04ff94e439d5d3dc261d5b44e)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.071015 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.078333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe2409-f1d0-4fa5-ba39-3fca713e9af1 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.084048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.085055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.085413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.085763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] PATH_INFO: `/v3/projects/39d74b57b467479f8fb7e0f44d937163` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.087139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39d74b57b467479f8fb7e0f44d937163)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.089169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.119772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:28.124325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81cf59b04ff94e439d5d3dc261d5b44e', 'request_id': 'req-3d0dcc77-04e1-474a-ab1e-a1f782546eec'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.131335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81cf59b04ff94e439d5d3dc261d5b44e', 'request_id': 'req-3d0dcc77-04e1-474a-ab1e-a1f782546eec'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.138545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81cf59b04ff94e439d5d3dc261d5b44e', 'request_id': 'req-3d0dcc77-04e1-474a-ab1e-a1f782546eec'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.140971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39d74b57b467479f8fb7e0f44d937163', 'request_id': 'req-a418abed-5465-4156-bc47-328ed9a79443'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.147231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39d74b57b467479f8fb7e0f44d937163', 'request_id': 'req-a418abed-5465-4156-bc47-328ed9a79443'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.151812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] The token cache is being invalidate because project 81cf59b04ff94e439d5d3dc261d5b44e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:28.152171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.153079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3d0dcc77-04e1-474a-ab1e-a1f782546eec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.154989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1150/2327] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:28 2022] DELETE /identity/v3/projects/81cf59b04ff94e439d5d3dc261d5b44e => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:28.156536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e1c7f9-d8aa-4808-8f9e-2388ec6f2ee8 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.157209 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e1c7f9-d8aa-4808-8f9e-2388ec6f2ee8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.157746 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e1c7f9-d8aa-4808-8f9e-2388ec6f2ee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.158249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39d74b57b467479f8fb7e0f44d937163', 'request_id': 'req-a418abed-5465-4156-bc47-328ed9a79443'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.170967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] The token cache is being invalidate because project 39d74b57b467479f8fb7e0f44d937163 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:28.171287 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.172138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.180135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1178/2328] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:27 2022] DELETE /identity/v3/projects/39d74b57b467479f8fb7e0f44d937163 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:28.180135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.180135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.182051 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a0e1c7f9-d8aa-4808-8f9e-2388ec6f2ee8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:28.184342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a0e1c7f9-d8aa-4808-8f9e-2388ec6f2ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.245204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:28.258989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1151/2329] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:28.260436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7820ca88-f6a0-42a9-857c-06df17f3cae6 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.260768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7820ca88-f6a0-42a9-857c-06df17f3cae6 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.261106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7820ca88-f6a0-42a9-857c-06df17f3cae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.269777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a418abed-5465-4156-bc47-328ed9a79443 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.274003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.274990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.275178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.275433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] PATH_INFO: `/v3/users/ac15d90ddb5d4cb8ba2a8523e999ccb9` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.276664 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac15d90ddb5d4cb8ba2a8523e999ccb9)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.278752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.280916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-7820ca88-f6a0-42a9-857c-06df17f3cae6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:28.282170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7820ca88-f6a0-42a9-857c-06df17f3cae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.291570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1152/2330] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:28.292579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7820ca88-f6a0-42a9-857c-06df17f3cae6 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.307400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7820ca88-f6a0-42a9-857c-06df17f3cae6 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.313927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116364ed-8958-47ea-9af5-bb9cb178c7be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.314627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116364ed-8958-47ea-9af5-bb9cb178c7be admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.314926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116364ed-8958-47ea-9af5-bb9cb178c7be admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.315304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116364ed-8958-47ea-9af5-bb9cb178c7be admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.316122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116364ed-8958-47ea-9af5-bb9cb178c7be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:28.316463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116364ed-8958-47ea-9af5-bb9cb178c7be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.318113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116364ed-8958-47ea-9af5-bb9cb178c7be admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.333693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1153/2331] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:28.334811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec381fb-27a8-4147-a749-1d236740e45b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.335114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec381fb-27a8-4147-a749-1d236740e45b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.335398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec381fb-27a8-4147-a749-1d236740e45b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.340138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac15d90ddb5d4cb8ba2a8523e999ccb9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.341093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac15d90ddb5d4cb8ba2a8523e999ccb9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.350569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac15d90ddb5d4cb8ba2a8523e999ccb9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.351668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac15d90ddb5d4cb8ba2a8523e999ccb9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.360352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-dec381fb-27a8-4147-a749-1d236740e45b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:28.367395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac15d90ddb5d4cb8ba2a8523e999ccb9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.378597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dec381fb-27a8-4147-a749-1d236740e45b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.395365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1179/2332] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:28 2022] DELETE /identity/v3/users/ac15d90ddb5d4cb8ba2a8523e999ccb9 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:28.396251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.405517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.415438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1154/2333] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:28.420630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec381fb-27a8-4147-a749-1d236740e45b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.432356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec381fb-27a8-4147-a749-1d236740e45b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.521795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.522912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.523130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec4425-bf6a-46a1-a45f-ef6a19576742 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.523306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.523665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.524453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:28.524956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.526618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.529390 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.530168 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.530585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.530987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.531785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.534031 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.548269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1155/2334] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:28.548269 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.550148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.568378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e55768c-81ca-45e7-aa2b-baddf6708f63 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.575953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.576890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.577171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.577453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] PATH_INFO: `/v3/users/4d23c3f5f358461da0b1d3ead1121f36` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.578748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d23c3f5f358461da0b1d3ead1121f36)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.583939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.586324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1180/2335] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/projects => generated 369 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:28.587639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.648665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:28.662097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d23c3f5f358461da0b1d3ead1121f36'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.663928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:28.671750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d23c3f5f358461da0b1d3ead1121f36'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.672832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d23c3f5f358461da0b1d3ead1121f36'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.673579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d23c3f5f358461da0b1d3ead1121f36'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.684122 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb886270-e3e8-4636-af64-9ed2aac0bda9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.695253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42382883-2ab0-4c16-8256-8f83720912cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.695945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42382883-2ab0-4c16-8256-8f83720912cc admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.696231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42382883-2ab0-4c16-8256-8f83720912cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.696546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42382883-2ab0-4c16-8256-8f83720912cc admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.698322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42382883-2ab0-4c16-8256-8f83720912cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.701397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42382883-2ab0-4c16-8256-8f83720912cc admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.716579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d23c3f5f358461da0b1d3ead1121f36'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:28.758088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1156/2336] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:28 2022] DELETE /identity/v3/users/4d23c3f5f358461da0b1d3ead1121f36 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:28.761009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.766022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.769225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1181/2337] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/projects => generated 383 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:28.774439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.774672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.774958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.800463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:28.801878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.864629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a13e49-b81f-473d-9164-cf62be4a7b82 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.910624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1182/2338] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:28.911579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.915522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.944518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86da7c1b-4cbe-46ca-b446-4d4826759488 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:28.949862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.951235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.951722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.952103 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.952959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.955396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.973964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:28.974763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:28.975301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:28.975407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:28.976139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:28.981063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:28.981212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1183/2339] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/users => generated 292 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:28.981212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:28.982063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:28.996714 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16105705-7eef-4f55-8dbf-d5040e8a45e6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.001779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1157/2340] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:28 2022] POST /identity/v3/users => generated 285 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:29.002202 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.003019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.003129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.003429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.003693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] PATH_INFO: `/v3/projects/248155f307ff41c4bf5be750535f8991` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.005388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] RBAC: Authorizing `identity:delete_project(project_id=248155f307ff41c4bf5be750535f8991)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.005936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.007335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.063314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '248155f307ff41c4bf5be750535f8991', 'request_id': 'req-5e239777-6105-41ad-a5b9-fb5aa570ecaa'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.066430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18cfd30-9c6a-4ece-9d91-653ebe4a87f8 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.070065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '248155f307ff41c4bf5be750535f8991', 'request_id': 'req-5e239777-6105-41ad-a5b9-fb5aa570ecaa'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.070217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:29.079811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '248155f307ff41c4bf5be750535f8991', 'request_id': 'req-5e239777-6105-41ad-a5b9-fb5aa570ecaa'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.095767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] The token cache is being invalidate because project 248155f307ff41c4bf5be750535f8991 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:29.096059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.096819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.102166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1184/2341] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:28 2022] DELETE /identity/v3/projects/248155f307ff41c4bf5be750535f8991 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:29.102166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.102166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.129777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36f37c4-1b8d-42bd-8d3d-ac5f1de6545f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.130433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f37c4-1b8d-42bd-8d3d-ac5f1de6545f service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.130801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f37c4-1b8d-42bd-8d3d-ac5f1de6545f service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.132088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36f37c4-1b8d-42bd-8d3d-ac5f1de6545f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.136159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f36f37c4-1b8d-42bd-8d3d-ac5f1de6545f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.152690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:29.176769 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e239777-6105-41ad-a5b9-fb5aa570ecaa admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.182333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.183138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.183432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.183712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.184536 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.186668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.197838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1185/2342] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:29.198672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.200413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.214495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f37c4-1b8d-42bd-8d3d-ac5f1de6545f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.214686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34985527-84bb-474f-b8fe-9593ae249438 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.216089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36f37c4-1b8d-42bd-8d3d-ac5f1de6545f service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.219497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.220171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.220496 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.221284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.221509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.224078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.226651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1158/2343] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:29.228027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a909ff-b4ce-4b54-8c29-9e7d6dbab3f5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.228291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a909ff-b4ce-4b54-8c29-9e7d6dbab3f5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.228609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a909ff-b4ce-4b54-8c29-9e7d6dbab3f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.246137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1186/2344] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:29.246137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.251557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c7a909ff-b4ce-4b54-8c29-9e7d6dbab3f5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:29.252994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c7a909ff-b4ce-4b54-8c29-9e7d6dbab3f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.258867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b740f4d-d197-4fcf-a672-e6aa76e96e80 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.264802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb09531a-1e16-4896-9ab2-4d985c3efc35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.265501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb09531a-1e16-4896-9ab2-4d985c3efc35 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.265789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb09531a-1e16-4896-9ab2-4d985c3efc35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.266065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb09531a-1e16-4896-9ab2-4d985c3efc35 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.266800 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb09531a-1e16-4896-9ab2-4d985c3efc35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.273986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1159/2345] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:29.274153 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb09531a-1e16-4896-9ab2-4d985c3efc35 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.274250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a909ff-b4ce-4b54-8c29-9e7d6dbab3f5 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.279986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a909ff-b4ce-4b54-8c29-9e7d6dbab3f5 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.280792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1187/2346] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:29.282101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1304605-0385-4ea9-ad9b-782a94d713ee None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.282389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1304605-0385-4ea9-ad9b-782a94d713ee None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.282677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1304605-0385-4ea9-ad9b-782a94d713ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.289530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.290314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.290603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.292023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.292598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.295122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.306142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e1304605-0385-4ea9-ad9b-782a94d713ee None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:29.307525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e1304605-0385-4ea9-ad9b-782a94d713ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.308976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1160/2347] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:29.310580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.322700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1188/2348] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:29.322700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1304605-0385-4ea9-ad9b-782a94d713ee None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.324864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ee9881-1430-4b9b-84ac-8aae3a19aac0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.332823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1304605-0385-4ea9-ad9b-782a94d713ee None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.332973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.333793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.334082 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.334361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] PATH_INFO: `/v3/projects/8f46cacb9afd497e8dc998789cf87b89/users/1ca21e157dba44fa934a880fa6cafedc/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.338585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19af712d-994e-4473-ac90-466e80d8f204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.340038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19af712d-994e-4473-ac90-466e80d8f204 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.340721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19af712d-994e-4473-ac90-466e80d8f204 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.341795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19af712d-994e-4473-ac90-466e80d8f204 admin admin] PATH_INFO: `/v3/projects/9a96afb5d6054ddcbf3b4cf54a2900b7/users/2d94be4d09b04c6b90c9fcfb6685546c/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.360685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f46cacb9afd497e8dc998789cf87b89, user_id=1ca21e157dba44fa934a880fa6cafedc, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.362526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19af712d-994e-4473-ac90-466e80d8f204 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a96afb5d6054ddcbf3b4cf54a2900b7, user_id=2d94be4d09b04c6b90c9fcfb6685546c, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.364827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.367694 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19af712d-994e-4473-ac90-466e80d8f204 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.385948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1161/2349] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:29 2022] PUT /identity/v3/projects/8f46cacb9afd497e8dc998789cf87b89/users/1ca21e157dba44fa934a880fa6cafedc/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:29.386931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.388740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1189/2350] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:29 2022] PUT /identity/v3/projects/9a96afb5d6054ddcbf3b4cf54a2900b7/users/2d94be4d09b04c6b90c9fcfb6685546c/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:29.396524 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398cf1ab-149e-427a-91a0-98579a2b7ae6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.397144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398cf1ab-149e-427a-91a0-98579a2b7ae6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.397617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398cf1ab-149e-427a-91a0-98579a2b7ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.439810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-398cf1ab-149e-427a-91a0-98579a2b7ae6 None None] MFA Rules not processed for user `1ca21e157dba44fa934a880fa6cafedc`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:29.441289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-398cf1ab-149e-427a-91a0-98579a2b7ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.527284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb42af35-8814-4865-a665-3c7004824c4f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.538659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1190/2351] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:29.538659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.538659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.538659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.549967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:29.588806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] MFA Rules not processed for user `2d94be4d09b04c6b90c9fcfb6685546c`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:29.592112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.615374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.616099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.616416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.616690 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.617443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:29.617760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.619420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.636439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1162/2352] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:29.636439 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.638167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.688123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1191/2353] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:29 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:29.689254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.693754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.708545 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35560549-4dd1-4359-be6e-42b0023aa2b4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.711616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b662f95-dde3-4a97-be62-e29020657b67 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.713635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.714598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.714983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.715395 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] PATH_INFO: `/v3/projects/42d627e0235a4a4caba84a4057275b5e` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.715585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.716810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.717097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.717379 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.717534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=42d627e0235a4a4caba84a4057275b5e)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.719784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.721996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:29.730522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.730522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.742426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1163/2354] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:29.743324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.755795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2b2b98-6f00-47f6-86b9-333387439d7c service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.763337 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.764246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.764568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.764854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.766257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:29.768571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:29.787142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42d627e0235a4a4caba84a4057275b5e', 'request_id': 'req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.795284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42d627e0235a4a4caba84a4057275b5e', 'request_id': 'req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.802361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1164/2355] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:29 2022] POST /identity/v3/projects => generated 371 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:29.803328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.805728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.806134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42d627e0235a4a4caba84a4057275b5e', 'request_id': 'req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.820708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] The token cache is being invalidate because project 42d627e0235a4a4caba84a4057275b5e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:29.821406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.822490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:29.824338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1192/2356] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:29 2022] DELETE /identity/v3/projects/42d627e0235a4a4caba84a4057275b5e => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:29.825541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:29.827167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.884873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a731c12-ef45-4cc6-93cf-2dc5b1b54392 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.890900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:29.891568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:29.892367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:29.892693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:29.895456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:29.923402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ccbdc0-1e01-434b-857a-9b12d7643f6e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:29.990218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:30.014542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.016147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.036459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1165/2357] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:29 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:30.049358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:30.049777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:30.050472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.050819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.051409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.051851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.053447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:30.054190 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.081219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb27c62-8b35-4549-abc9-b153513dccd4 service glance] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:30.085488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1193/2358] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:29 2022] POST /identity/v3/users => generated 286 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:30.089019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:30.100961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28e40f-b0b5-4319-9ca0-fabfb637e363 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:30.106882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:30.107788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:30.107929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.108246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.108705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.109737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.110735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.111885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.112862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.114290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] PATH_INFO: `/v3/users/99a8143822414ba9968f712e776586fa` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.122571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=99a8143822414ba9968f712e776586fa)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.125669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.126554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1194/2359] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:30.131162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:30.146492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec763128-59ff-4406-ac7e-2c84b71c24ef admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:30.151778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:30.152655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.152840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.153155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.154667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.156429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.169170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1195/2360] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:30.173126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:30.181983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:30.187532 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a03b8f4-4f89-4ca1-abea-453ceb3d7712 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:30.194382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2fb085-2dbe-4a29-bd5d-1a953c994864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:30.195900 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2fb085-2dbe-4a29-bd5d-1a953c994864 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.196460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2fb085-2dbe-4a29-bd5d-1a953c994864 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.197233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2fb085-2dbe-4a29-bd5d-1a953c994864 admin admin] PATH_INFO: `/v3/projects/9edab973767545988e178ddff76557ac/users/7e322227b42b4aaa99674afd906a706a/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.219038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2fb085-2dbe-4a29-bd5d-1a953c994864 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9edab973767545988e178ddff76557ac, user_id=7e322227b42b4aaa99674afd906a706a, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.222365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2fb085-2dbe-4a29-bd5d-1a953c994864 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.266008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1196/2361] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:30 2022] PUT /identity/v3/projects/9edab973767545988e178ddff76557ac/users/7e322227b42b4aaa99674afd906a706a/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:30.277141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.280329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.280680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.304601 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99a8143822414ba9968f712e776586fa'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.317334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99a8143822414ba9968f712e776586fa'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.318315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99a8143822414ba9968f712e776586fa'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.319283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:30.320788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99a8143822414ba9968f712e776586fa'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.322303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:30.373398 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99a8143822414ba9968f712e776586fa'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.401185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1166/2362] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:30 2022] DELETE /identity/v3/users/99a8143822414ba9968f712e776586fa => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:30.476817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1197/2363] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:30.493020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:30.494661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:30.576787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3d90ea-3b97-4f79-8c8c-2637a446bdbb admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:30.585995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:30.586915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.587309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.587699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.632267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:30.634432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:30.650503 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c827b821-c941-4b30-b7e5-af275e205f70 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:30.692765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.695203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.711119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1167/2364] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:30.741737 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:30.742994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:30.743445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:30.743927 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] PATH_INFO: `/v3/users/45838beec30e44b8ac45849c517dcd20` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:30.745830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=45838beec30e44b8ac45849c517dcd20)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:30.749777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:30.845864 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45838beec30e44b8ac45849c517dcd20'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.846702 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45838beec30e44b8ac45849c517dcd20'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.854795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45838beec30e44b8ac45849c517dcd20'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.855539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45838beec30e44b8ac45849c517dcd20'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.866859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45838beec30e44b8ac45849c517dcd20'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:30.888887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:30.897716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1198/2365] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:30 2022] DELETE /identity/v3/users/45838beec30e44b8ac45849c517dcd20 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:31.025132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:31.026548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:31.044860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0105b99e-418d-4c4e-b79b-0484f19e954a service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:31.061871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:31.140749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:31.141653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:31.141986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:31.142315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] PATH_INFO: `/v3/projects/a3d80ef13e1c4fa7b79d25f5e04a7fa4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:31.143523 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3d80ef13e1c4fa7b79d25f5e04a7fa4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:31.145490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:31.202444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:31.206098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:31.252867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3d80ef13e1c4fa7b79d25f5e04a7fa4', 'request_id': 'req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.263567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3d80ef13e1c4fa7b79d25f5e04a7fa4', 'request_id': 'req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.271576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa0bedd-5118-4275-887e-d0438fa7fb38 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:31.277020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3d80ef13e1c4fa7b79d25f5e04a7fa4', 'request_id': 'req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.321659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] The token cache is being invalidate because project a3d80ef13e1c4fa7b79d25f5e04a7fa4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:31.321659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.322643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.327548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1168/2366] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:31 2022] DELETE /identity/v3/projects/a3d80ef13e1c4fa7b79d25f5e04a7fa4 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:31.332057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:31.336233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:31.414946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:31.415680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:31.415959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:31.416265 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] PATH_INFO: `/v3/projects/0cc71645fab147959f329fe030163996` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:31.417458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0cc71645fab147959f329fe030163996)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:31.419417 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:31.473973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcff6ea-c1ad-4aac-8e01-b0e9c9f49651 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:31.479756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:31.480788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:31.481252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:31.481664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] PATH_INFO: `/v3/users/ebdfcd8dca69452ca8a85aaabef64b48` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:31.483340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ebdfcd8dca69452ca8a85aaabef64b48)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:31.485595 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:31.510078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0cc71645fab147959f329fe030163996', 'request_id': 'req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.519930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0cc71645fab147959f329fe030163996', 'request_id': 'req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.541591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0cc71645fab147959f329fe030163996', 'request_id': 'req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.557516 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:31.560155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ebdfcd8dca69452ca8a85aaabef64b48'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.573609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] The token cache is being invalidate because project 0cc71645fab147959f329fe030163996 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:31.574930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.578698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.581818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ebdfcd8dca69452ca8a85aaabef64b48'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.585187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ebdfcd8dca69452ca8a85aaabef64b48'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.585680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1199/2367] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:31 2022] DELETE /identity/v3/projects/0cc71645fab147959f329fe030163996 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:31.585767 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ebdfcd8dca69452ca8a85aaabef64b48'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.586828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:31.596204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:31.631732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ebdfcd8dca69452ca8a85aaabef64b48'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.659391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1169/2368] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:31 2022] DELETE /identity/v3/users/ebdfcd8dca69452ca8a85aaabef64b48 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:31.756294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9e15ae-e7a0-4e4f-9608-85b6d3e3ec02 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:31.815855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:31.868920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:31.870562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:31.870850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:31.871121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] PATH_INFO: `/v3/users/5b875d49366e4906937507e0b68292a3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:31.872943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b875d49366e4906937507e0b68292a3)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:31.874884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:31.948272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b875d49366e4906937507e0b68292a3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.949076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b875d49366e4906937507e0b68292a3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.959731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b875d49366e4906937507e0b68292a3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.961263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b875d49366e4906937507e0b68292a3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.970421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b875d49366e4906937507e0b68292a3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:31.992204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1200/2369] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:31 2022] DELETE /identity/v3/users/5b875d49366e4906937507e0b68292a3 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:32.261590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:32.265588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:32.286430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0a0c3-4fbe-4ca8-bbd6-6f2ff71465a9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:32.326523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:32.328285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:32.429339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:32.430898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.431440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.432040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] PATH_INFO: `/v3/projects/14d94ce0ce5e486c80378acc134168f7` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.433675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=14d94ce0ce5e486c80378acc134168f7)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:32.436749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:32.447464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f337733f-6813-4792-b918-d6be8a8553f4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:32.452904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:32.454028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.454506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.454789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] PATH_INFO: `/v3/projects/5350d7b1d1f54c8880b1c2f6c2d1c31a` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.456566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=5350d7b1d1f54c8880b1c2f6c2d1c31a)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:32.458843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:32.499494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14d94ce0ce5e486c80378acc134168f7', 'request_id': 'req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.506727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14d94ce0ce5e486c80378acc134168f7', 'request_id': 'req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.515028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14d94ce0ce5e486c80378acc134168f7', 'request_id': 'req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.519204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5350d7b1d1f54c8880b1c2f6c2d1c31a', 'request_id': 'req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.528162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5350d7b1d1f54c8880b1c2f6c2d1c31a', 'request_id': 'req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.531105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:32.537150 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] The token cache is being invalidate because project 14d94ce0ce5e486c80378acc134168f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:32.537598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.538331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3b3c7f5a-5e66-49c4-b0c6-6f753f50de2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.540915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5350d7b1d1f54c8880b1c2f6c2d1c31a', 'request_id': 'req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.541103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1170/2370] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:32 2022] DELETE /identity/v3/projects/14d94ce0ce5e486c80378acc134168f7 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:32.546651 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4c01f-9b3c-4514-a1cf-d6ddfb3e8d00 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.550574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4c01f-9b3c-4514-a1cf-d6ddfb3e8d00 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.550718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4c01f-9b3c-4514-a1cf-d6ddfb3e8d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.560586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] The token cache is being invalidate because project 5350d7b1d1f54c8880b1c2f6c2d1c31a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:32.561916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.562767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-67c1014e-4ed2-4b24-b7c5-299f0541dc1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:32.569054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1201/2371] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:32 2022] DELETE /identity/v3/projects/5350d7b1d1f54c8880b1c2f6c2d1c31a => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:32.572029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5377fe73-546f-4a40-8750-11b53cddaad6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.572587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5377fe73-546f-4a40-8750-11b53cddaad6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.573121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5377fe73-546f-4a40-8750-11b53cddaad6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.579017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-23c4c01f-9b3c-4514-a1cf-d6ddfb3e8d00 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:32.580844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-23c4c01f-9b3c-4514-a1cf-d6ddfb3e8d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:32.609642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5377fe73-546f-4a40-8750-11b53cddaad6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:32.611050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5377fe73-546f-4a40-8750-11b53cddaad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:32.695804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:32.717840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1171/2372] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:32.727870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60852e6-2d33-49db-ab90-9769ef5ab84c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.728156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60852e6-2d33-49db-ab90-9769ef5ab84c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.728871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60852e6-2d33-49db-ab90-9769ef5ab84c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.769383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c60852e6-2d33-49db-ab90-9769ef5ab84c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:32.771236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1202/2373] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:32.772777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c54d7-dfa3-4f99-a708-3f092b8f71ec None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.773061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c54d7-dfa3-4f99-a708-3f092b8f71ec None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.773346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c54d7-dfa3-4f99-a708-3f092b8f71ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.773689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c60852e6-2d33-49db-ab90-9769ef5ab84c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:32.793472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1172/2374] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:32.798225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60852e6-2d33-49db-ab90-9769ef5ab84c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:32.806067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-437c54d7-dfa3-4f99-a708-3f092b8f71ec None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:32.807313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-437c54d7-dfa3-4f99-a708-3f092b8f71ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:32.815023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60852e6-2d33-49db-ab90-9769ef5ab84c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:32.828214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:32.829074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.829450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.829854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.832084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:32.832731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:32.839839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:32.849970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1203/2375] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:32.852744 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437c54d7-dfa3-4f99-a708-3f092b8f71ec None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:32.875542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437c54d7-dfa3-4f99-a708-3f092b8f71ec None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:32.876501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1173/2376] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:32.882409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:32.895041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:32.897966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.898373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.899509 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.910773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:32.911010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8125c3d-7a5d-4bc7-974c-9f2763a0f0fd admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:32.911523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:32.913437 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:32.934736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:32.936314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:32.936755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:32.937802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:32.937802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:32.939530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:32.960124 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1204/2377] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:32.961105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:32.997865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1174/2378] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:32 2022] POST /identity/v3/projects => generated 375 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:32.999104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.005310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d88093-1bac-4a49-9ea3-ac6b8d1b9d59 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.140784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e552ed-127b-4b38-b7a1-7c7ba052f87c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.141866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.142792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.143150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.143602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.144378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.146499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.146730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.147574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.148020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.148495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.149338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.151449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.189531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1175/2379] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:32 2022] POST /identity/v3/users => generated 288 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:33.194599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.213229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1205/2380] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:32 2022] POST /identity/v3/projects => generated 373 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:33.215331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f211b9-0480-438a-9cd3-059c032eb737 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.217855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.318102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:INSERT=2,keystone:DELETE=1,keystone:SELECT=97 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:33.360877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dc1ab1-272a-4db5-a61b-860c0a98745e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.361092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.361786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.362075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.362448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.363723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.365404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.367121 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.367818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.368263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.368733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.369779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.371757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.378286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1176/2381] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:33 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:33.385408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.409536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4645aa94-8251-42a5-8bdc-e842c0b894dc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.419204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1206/2382] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:33 2022] POST /identity/v3/users => generated 287 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:33.424290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.426878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.427770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.428411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.428863 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.431547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.434243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.443862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce049862-b4bb-4fe7-8727-9aad6c42685c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.448147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1177/2383] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:33 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:33.450056 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.451463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.451839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.452316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.453142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.453571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.457626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.467000 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84368a4d-9315-4a97-b2d8-4bbb0091533a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.476136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a9ddcec-a4f4-4c7b-b811-af74b076043c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.477351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9ddcec-a4f4-4c7b-b811-af74b076043c admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.477850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9ddcec-a4f4-4c7b-b811-af74b076043c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.478297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a9ddcec-a4f4-4c7b-b811-af74b076043c admin admin] PATH_INFO: `/v3/projects/02642b99cd0948538503f2257be2c33f/users/a401503e585c47fe94ba5d243d090f8b/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.482126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1207/2384] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:33 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:33.484832 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.506644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542eddb-f356-4ce7-b60a-2ab08b8a95c9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.510893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9ddcec-a4f4-4c7b-b811-af74b076043c admin admin] RBAC: Authorizing `identity:create_grant(project_id=02642b99cd0948538503f2257be2c33f, user_id=a401503e585c47fe94ba5d243d090f8b, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.513665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.514516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.514803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.515076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.515801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.515936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a9ddcec-a4f4-4c7b-b811-af74b076043c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.517493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.534090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1208/2385] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:33 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:33.548200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.557339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1178/2386] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:33 2022] PUT /identity/v3/projects/02642b99cd0948538503f2257be2c33f/users/a401503e585c47fe94ba5d243d090f8b/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:33.564301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.565633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.566035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.574716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:33.574861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adf055f-5d17-4cd2-972e-47384e0bef6c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.622765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] MFA Rules not processed for user `a401503e585c47fe94ba5d243d090f8b`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:33.624096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:33.733252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1179/2387] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:33 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:33.739456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96720984-dc03-44c3-8609-b8ecb19adb26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.740299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96720984-dc03-44c3-8609-b8ecb19adb26 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.740932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96720984-dc03-44c3-8609-b8ecb19adb26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.740932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96720984-dc03-44c3-8609-b8ecb19adb26 admin admin] PATH_INFO: `/v3/projects/9bf4c2e6ff4347f5b5e46efe5d431dc9/users/aa8cee3ae343415aae94ce2592a04dd3/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.749271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:33.751105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:33.770512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96720984-dc03-44c3-8609-b8ecb19adb26 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9bf4c2e6ff4347f5b5e46efe5d431dc9, user_id=aa8cee3ae343415aae94ce2592a04dd3, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:33.773730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96720984-dc03-44c3-8609-b8ecb19adb26 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:33.794907 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1209/2388] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:33 2022] PUT /identity/v3/projects/9bf4c2e6ff4347f5b5e46efe5d431dc9/users/aa8cee3ae343415aae94ce2592a04dd3/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:33.857128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5784c2b9-138e-4fb8-9ef3-dd27110dc1af None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:33.875548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.875876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.876156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:33.914990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] MFA Rules not processed for user `aa8cee3ae343415aae94ce2592a04dd3`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:33.917715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:33.926825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26eaa2e-b68b-4bc8-abf5-033c168e6054 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:33.927682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26eaa2e-b68b-4bc8-abf5-033c168e6054 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:33.928416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26eaa2e-b68b-4bc8-abf5-033c168e6054 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:33.928657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26eaa2e-b68b-4bc8-abf5-033c168e6054 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:34.022428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26eaa2e-b68b-4bc8-abf5-033c168e6054 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:34.025113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26eaa2e-b68b-4bc8-abf5-033c168e6054 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:34.025917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1210/2389] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:33 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:34.040459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1180/2390] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:33 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:12:34.117084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:34.118518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:34.131347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e57ef0-1bbb-474c-b4b5-29ea7bf776de None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:34.136332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:34.137368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:34.137655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:34.139184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:34.155192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:34.156693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:34.181147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1211/2391] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:34 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:36.516910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db19fc8f-6b63-4ad2-8466-5a35a1264592 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:36.517469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db19fc8f-6b63-4ad2-8466-5a35a1264592 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:36.517469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db19fc8f-6b63-4ad2-8466-5a35a1264592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:36.524843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:36.547386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-db19fc8f-6b63-4ad2-8466-5a35a1264592 None None] MFA Rules not processed for user `7e322227b42b4aaa99674afd906a706a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:36.549473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-db19fc8f-6b63-4ad2-8466-5a35a1264592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:36.625061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1181/2392] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:36 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:36.629285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:36.631310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:36.660116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:36.713148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf01df8-4fcc-4dbf-af86-d6b053642217 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:36.718607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:36.719281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:36.719560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:36.719834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:36.720584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:36.722510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:36.755787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1212/2393] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:36 2022] POST /identity/v3/projects => generated 371 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:36.757617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db19fc8f-6b63-4ad2-8466-5a35a1264592 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:36.770781 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db19fc8f-6b63-4ad2-8466-5a35a1264592 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:36.864965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:36.865788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:36.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:36.866351 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:36.867128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:36.869382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:36.897162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1182/2394] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:36 2022] POST /identity/v3/users => generated 292 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:36.898995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:36.920027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2f7a0-468c-4acb-a14c-5d1f4eeeaaa0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:36.928901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:36.929734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:36.930698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:36.931086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:36.931939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:36.933873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:36.946045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1213/2395] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:36 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:36.950013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:36.961440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a868c09b-28dd-455d-8220-bb970a67d452 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:36.966808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:36.967794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:36.968149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:36.968553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] PATH_INFO: `/v3/projects/21761824bf0a46f996a66abd2b210e6d/users/80c376e37e9041e79ac92e72f4d71468/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:36.986171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=21761824bf0a46f996a66abd2b210e6d, user_id=80c376e37e9041e79ac92e72f4d71468, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:36.989189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:37.009424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1183/2396] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:36 2022] PUT /identity/v3/projects/21761824bf0a46f996a66abd2b210e6d/users/80c376e37e9041e79ac92e72f4d71468/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:37.010725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:37.024039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67d9e03-cb68-4818-bb23-0c21cee09a50 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:37.085830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ab64b2-c667-4dbe-aa11-a85aa79efd31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:37.086513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab64b2-c667-4dbe-aa11-a85aa79efd31 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:37.086796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab64b2-c667-4dbe-aa11-a85aa79efd31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:37.087071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab64b2-c667-4dbe-aa11-a85aa79efd31 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:37.087991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ab64b2-c667-4dbe-aa11-a85aa79efd31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:37.089675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ab64b2-c667-4dbe-aa11-a85aa79efd31 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:37.102515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1214/2397] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:37 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:37.106775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:37.117837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f9dd7b-d011-44db-b17f-8a93c46d2d3d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:37.123760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:37.124881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:37.125832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:37.125832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] PATH_INFO: `/v3/projects/21761824bf0a46f996a66abd2b210e6d/users/80c376e37e9041e79ac92e72f4d71468/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:37.127623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21761824bf0a46f996a66abd2b210e6d, user_id=80c376e37e9041e79ac92e72f4d71468, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:37.130915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:37.146218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1184/2398] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:37 2022] PUT /identity/v3/projects/21761824bf0a46f996a66abd2b210e6d/users/80c376e37e9041e79ac92e72f4d71468/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:42.069822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5525d972-56c4-400b-b990-4255b61a3ffa None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:42.070836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5525d972-56c4-400b-b990-4255b61a3ffa None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:42.071425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5525d972-56c4-400b-b990-4255b61a3ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:42.102160 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5525d972-56c4-400b-b990-4255b61a3ffa None None] MFA Rules not processed for user `80c376e37e9041e79ac92e72f4d71468`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:42.103538 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5525d972-56c4-400b-b990-4255b61a3ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:42.179601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1215/2399] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:42 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:42.193239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:42.194970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:42.250642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:42.284476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e69fa72-8e74-4903-884f-0f3586b95509 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:42.289885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b6be34-c2a3-40a2-a3ef-1fb2c42a27d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:42.290608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b6be34-c2a3-40a2-a3ef-1fb2c42a27d1 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:42.290960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b6be34-c2a3-40a2-a3ef-1fb2c42a27d1 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:42.291302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b6be34-c2a3-40a2-a3ef-1fb2c42a27d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:42.347026 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b6be34-c2a3-40a2-a3ef-1fb2c42a27d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:42.348656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b6be34-c2a3-40a2-a3ef-1fb2c42a27d1 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:42.358353 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1185/2400] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:43.570740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f30a9-548b-423d-80b2-84bdc90483ff None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:43.571251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f30a9-548b-423d-80b2-84bdc90483ff None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:43.571392 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378f30a9-548b-423d-80b2-84bdc90483ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:43.590405 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-378f30a9-548b-423d-80b2-84bdc90483ff None None] MFA Rules not processed for user `7e322227b42b4aaa99674afd906a706a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:43.591529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-378f30a9-548b-423d-80b2-84bdc90483ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:43.598505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1216/2401] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:43 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:46.344766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51b1452-ff15-476b-bd45-7937e648d623 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:46.345580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51b1452-ff15-476b-bd45-7937e648d623 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:46.346020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51b1452-ff15-476b-bd45-7937e648d623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:46.377460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f51b1452-ff15-476b-bd45-7937e648d623 None None] MFA Rules not processed for user `aa8cee3ae343415aae94ce2592a04dd3`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:46.379085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f51b1452-ff15-476b-bd45-7937e648d623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:46.478580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1186/2402] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:46 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:48.043911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378f30a9-548b-423d-80b2-84bdc90483ff None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:48.055369 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:48.170035 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378f30a9-548b-423d-80b2-84bdc90483ff None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:48.175511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:48.176567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:48.176925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:48.177274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] PATH_INFO: `/v3/users/a401503e585c47fe94ba5d243d090f8b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:48.178839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a401503e585c47fe94ba5d243d090f8b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:48.181988 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:48.249636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a401503e585c47fe94ba5d243d090f8b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.250470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a401503e585c47fe94ba5d243d090f8b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.260841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a401503e585c47fe94ba5d243d090f8b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.260841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a401503e585c47fe94ba5d243d090f8b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.269307 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a401503e585c47fe94ba5d243d090f8b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.298119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1217/2403] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:48 2022] DELETE /identity/v3/users/a401503e585c47fe94ba5d243d090f8b => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:48.301044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:48.301878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:48.302328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:48.344810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:48.346040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:48.388123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:48.410370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1187/2404] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:48.419709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:48.421149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:48.481269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:48.529957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30910734-88e1-40d1-9551-6a316232a778 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:48.534206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70201220-ff51-4406-a7fa-5d073554fd8f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:48.534858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70201220-ff51-4406-a7fa-5d073554fd8f service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:48.535144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70201220-ff51-4406-a7fa-5d073554fd8f service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:48.535421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70201220-ff51-4406-a7fa-5d073554fd8f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:48.545115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70201220-ff51-4406-a7fa-5d073554fd8f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:48.547328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70201220-ff51-4406-a7fa-5d073554fd8f service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:48.557558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1218/2405] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:48.597892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:48.599316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:48.611602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2345703-46cc-4c0d-99ed-9b8901befb14 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:48.616679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:48.617572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:48.617861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:48.618143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] PATH_INFO: `/v3/users/1ca21e157dba44fa934a880fa6cafedc` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:48.619363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ca21e157dba44fa934a880fa6cafedc)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:48.621480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:48.685418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ca21e157dba44fa934a880fa6cafedc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.694145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ca21e157dba44fa934a880fa6cafedc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.695169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ca21e157dba44fa934a880fa6cafedc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.695519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ca21e157dba44fa934a880fa6cafedc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.715837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ca21e157dba44fa934a880fa6cafedc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.737110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1188/2406] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:48 2022] DELETE /identity/v3/users/1ca21e157dba44fa934a880fa6cafedc => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:48.744971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:48.745272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:48.745563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:48.765165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:48.772453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:48.851211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1219/2407] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:48.861132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:48.864153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:48.882946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:48.884419 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:48.897241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19089454-1cae-4437-95b9-d8cbbae2654b None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:48.902259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:48.903125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:48.903457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:48.903734 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] PATH_INFO: `/v3/projects/02642b99cd0948538503f2257be2c33f` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:48.905178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02642b99cd0948538503f2257be2c33f)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:48.911474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:48.964368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbeb8fc-46c0-4397-90f1-ac85302c22b5 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:48.973083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02642b99cd0948538503f2257be2c33f', 'request_id': 'req-c7414683-d68e-4a83-9de2-9618c3f75840'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.980612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02642b99cd0948538503f2257be2c33f', 'request_id': 'req-c7414683-d68e-4a83-9de2-9618c3f75840'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:48.991489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02642b99cd0948538503f2257be2c33f', 'request_id': 'req-c7414683-d68e-4a83-9de2-9618c3f75840'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.005243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] The token cache is being invalidate because project 02642b99cd0948538503f2257be2c33f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:49.005566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.006232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c7414683-d68e-4a83-9de2-9618c3f75840 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.007841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1220/2408] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:48 2022] DELETE /identity/v3/projects/02642b99cd0948538503f2257be2c33f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:49.012626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab4f4d-d6df-4b7b-a2b9-2e4f666e93c8 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.012920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab4f4d-d6df-4b7b-a2b9-2e4f666e93c8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.013204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ab4f4d-d6df-4b7b-a2b9-2e4f666e93c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.028904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec65a2b-c086-4748-8232-c4bc14b4b061 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.029681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec65a2b-c086-4748-8232-c4bc14b4b061 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.030264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec65a2b-c086-4748-8232-c4bc14b4b061 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.030621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec65a2b-c086-4748-8232-c4bc14b4b061 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.033642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2ec65a2b-c086-4748-8232-c4bc14b4b061 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.052349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d4ab4f4d-d6df-4b7b-a2b9-2e4f666e93c8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:49.053569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d4ab4f4d-d6df-4b7b-a2b9-2e4f666e93c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.061014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:49.072606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:49.106918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec65a2b-c086-4748-8232-c4bc14b4b061 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.108643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec65a2b-c086-4748-8232-c4bc14b4b061 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.119392 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1189/2409] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:49.126381 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1221/2410] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:49.134183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723e984-7f76-4e5a-8298-62e571654e49 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.134473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723e984-7f76-4e5a-8298-62e571654e49 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.134786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1723e984-7f76-4e5a-8298-62e571654e49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.153736 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1723e984-7f76-4e5a-8298-62e571654e49 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:49.155225 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1723e984-7f76-4e5a-8298-62e571654e49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.168369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1190/2411] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:49.172852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab4f4d-d6df-4b7b-a2b9-2e4f666e93c8 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.184515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ab4f4d-d6df-4b7b-a2b9-2e4f666e93c8 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.190320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.190950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.191297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.191529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.192240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:49.192611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.194282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.211991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1222/2412] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:12:49.216671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1723e984-7f76-4e5a-8298-62e571654e49 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.227253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1723e984-7f76-4e5a-8298-62e571654e49 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.233037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.233696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.233987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.234296 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.235024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.237002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.264754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1191/2413] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:49.269672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.280030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c633bd-6a9f-4c5f-b294-6b241e075bd5 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.357099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.358208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.358490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.358759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.359958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.361889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.387362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1223/2414] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/users => generated 293 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:49.391454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.404257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc753e7-832f-4349-ab02-5ecacb0b38b1 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.410986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.411662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.411955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.412234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.413008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.414624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.425778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.426874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1192/2415] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:49.428420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.430688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.455372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98524bc7-97e6-4ca2-b397-a682674f243a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.462671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.463723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.463908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112b8053-bd39-4b23-83d9-70e647cd2341 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.464030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.464279 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] PATH_INFO: `/v3/projects/8f46cacb9afd497e8dc998789cf87b89` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.466335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f46cacb9afd497e8dc998789cf87b89)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.468805 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.469496 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.470479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.470802 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.471861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.472189 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.474010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.488689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1193/2416] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:49.493144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.508223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12bfbc4-979a-4022-bf38-216a34482344 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.513563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97869e2f-2a49-437e-a5a1-9b807a2d3492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.514470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97869e2f-2a49-437e-a5a1-9b807a2d3492 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.514753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97869e2f-2a49-437e-a5a1-9b807a2d3492 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.515032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97869e2f-2a49-437e-a5a1-9b807a2d3492 admin admin] PATH_INFO: `/v3/projects/4c65da84ed2443fbbeae23f12915eb0c/users/98288db85c6f43519ce433dd0a8e3b6b/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.532246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97869e2f-2a49-437e-a5a1-9b807a2d3492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c65da84ed2443fbbeae23f12915eb0c, user_id=98288db85c6f43519ce433dd0a8e3b6b, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.533206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f46cacb9afd497e8dc998789cf87b89', 'request_id': 'req-bf71a26c-f975-483f-a219-985a27209cbf'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.535828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97869e2f-2a49-437e-a5a1-9b807a2d3492 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.539972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f46cacb9afd497e8dc998789cf87b89', 'request_id': 'req-bf71a26c-f975-483f-a219-985a27209cbf'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.551083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f46cacb9afd497e8dc998789cf87b89', 'request_id': 'req-bf71a26c-f975-483f-a219-985a27209cbf'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.552098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1194/2417] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:49 2022] PUT /identity/v3/projects/4c65da84ed2443fbbeae23f12915eb0c/users/98288db85c6f43519ce433dd0a8e3b6b/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:12:49.570584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] The token cache is being invalidate because project 8f46cacb9afd497e8dc998789cf87b89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:49.571312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.572138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-bf71a26c-f975-483f-a219-985a27209cbf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:49.574166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1224/2418] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:49 2022] DELETE /identity/v3/projects/8f46cacb9afd497e8dc998789cf87b89 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:49.587123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb4190-602c-4ef9-b957-a4b448eaa0f2 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.587123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb4190-602c-4ef9-b957-a4b448eaa0f2 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.588803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb4190-602c-4ef9-b957-a4b448eaa0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.609135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-92bb4190-602c-4ef9-b957-a4b448eaa0f2 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:49.610832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-92bb4190-602c-4ef9-b957-a4b448eaa0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.622262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.622554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.622834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.656629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] MFA Rules not processed for user `98288db85c6f43519ce433dd0a8e3b6b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:49.657844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.667006 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:49.684211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1195/2419] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:49.693464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc36b8-22f2-4ebf-a857-84fc58d4825b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.693872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc36b8-22f2-4ebf-a857-84fc58d4825b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.694279 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccc36b8-22f2-4ebf-a857-84fc58d4825b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.712625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1ccc36b8-22f2-4ebf-a857-84fc58d4825b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:49.713832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1ccc36b8-22f2-4ebf-a857-84fc58d4825b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.722600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1196/2420] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:49.726817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccc36b8-22f2-4ebf-a857-84fc58d4825b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.733210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1225/2421] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:49.739365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccc36b8-22f2-4ebf-a857-84fc58d4825b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.745781 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.747217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:49.752326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.753211 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.753500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.753782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.754521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:12:49.754831 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.756489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.772628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1197/2422] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:12:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:12:49.773539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.783821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49606af6-758b-4d32-aba7-8dd74423e220 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.788622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.789288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.789582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.789853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.790569 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.792585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.815686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52135893-1c27-4d50-8f91-d59419fc8ada None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.819570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.820228 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.820573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.820847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.821776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1198/2423] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/projects => generated 375 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:12:49.826631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.830036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.831784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:49.841253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1226/2424] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:49 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:49.896291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c30e55-ce49-4870-b132-e9f938516e83 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.901088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.901768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.902058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.902341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.903063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.905173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.928432 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1199/2425] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:12:49 2022] POST /identity/v3/users => generated 288 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:12:49.931376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.947789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251dce4e-fed4-4fb5-85ec-591f7e858f59 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.952456 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.953163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.953460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.953740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.954470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.956075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:49.967113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1227/2426] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:12:49.971587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:49.983764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e6db6f-973a-4866-8599-64c784408d7c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:49.990460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d639eef8-b0a0-44d5-b090-a0b9845585a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:49.991215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d639eef8-b0a0-44d5-b090-a0b9845585a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:49.991563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d639eef8-b0a0-44d5-b090-a0b9845585a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:49.991829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d639eef8-b0a0-44d5-b090-a0b9845585a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:49.992632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d639eef8-b0a0-44d5-b090-a0b9845585a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:49.994307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d639eef8-b0a0-44d5-b090-a0b9845585a3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:50.009406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1200/2427] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:12:49 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:12:50.013916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:50.026309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7de833-8211-4474-a168-267519d7f876 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:50.031727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:50.032661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:50.032947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:50.033222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] PATH_INFO: `/v3/projects/ae4ab41d077c49d1b7181d7b78618a6f/users/df9113517b284ee1911b73b108dcd5bc/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:50.052515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae4ab41d077c49d1b7181d7b78618a6f, user_id=df9113517b284ee1911b73b108dcd5bc, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:50.055424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:50.074041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1228/2428] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:12:50 2022] PUT /identity/v3/projects/ae4ab41d077c49d1b7181d7b78618a6f/users/df9113517b284ee1911b73b108dcd5bc/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:50.085371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:50.085939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:50.086212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:50.124238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] MFA Rules not processed for user `df9113517b284ee1911b73b108dcd5bc`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:50.124238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:50.212793 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1201/2429] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:50 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:50.225472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:50.279843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cc965a-27c5-4cc6-8e4d-df894eccfeca admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:50.284053 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d37fd8-ad51-4ead-86e2-fd76e9e1fac3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:50.284718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d37fd8-ad51-4ead-86e2-fd76e9e1fac3 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:50.285005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d37fd8-ad51-4ead-86e2-fd76e9e1fac3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:50.285280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d37fd8-ad51-4ead-86e2-fd76e9e1fac3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:50.293521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d37fd8-ad51-4ead-86e2-fd76e9e1fac3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:50.294944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d37fd8-ad51-4ead-86e2-fd76e9e1fac3 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:50.303697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1229/2430] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:50 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:52.424557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:52.426123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:52.501990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6f5be3-5757-4a07-af95-bbae89c0e9f9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:52.507090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:52.507964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:52.508257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:52.508566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] PATH_INFO: `/v3/users/98288db85c6f43519ce433dd0a8e3b6b` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:52.509774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] RBAC: Authorizing `identity:delete_user(user_id=98288db85c6f43519ce433dd0a8e3b6b)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:52.511715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:52.527371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:52.562596 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98288db85c6f43519ce433dd0a8e3b6b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:52.570047 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98288db85c6f43519ce433dd0a8e3b6b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:52.570708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98288db85c6f43519ce433dd0a8e3b6b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:52.571314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98288db85c6f43519ce433dd0a8e3b6b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:52.583786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98288db85c6f43519ce433dd0a8e3b6b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:52.598850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1202/2431] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:12:52 2022] DELETE /identity/v3/users/98288db85c6f43519ce433dd0a8e3b6b => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:12:52.603389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34bc4c-69d1-4742-a66c-c9e19debc464 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:52.603853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34bc4c-69d1-4742-a66c-c9e19debc464 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:52.603983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34bc4c-69d1-4742-a66c-c9e19debc464 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:52.622345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0e34bc4c-69d1-4742-a66c-c9e19debc464 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:52.623547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0e34bc4c-69d1-4742-a66c-c9e19debc464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:52.639650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:12:52.691464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1230/2432] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:12:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:12:52.702897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:52.704334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:52.774590 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627e66d5-7c5e-4b76-9f22-9a41edd30dca admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:52.779603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:52.780409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:52.780787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:52.781056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:52.793852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:52.795327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:52.807475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1203/2433] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:12:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:12:52.981083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:52.981537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:52.981893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:53.020708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:12:53.021877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:53.085691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1231/2434] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:12:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:12:53.134020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:53.135585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:53.237532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0711a517-eef1-4c30-9920-282ed80cd0c4 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:53.239687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:12:53.241943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:12:53.252444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14fd632-2f71-456e-bb5c-5d3574a7adc0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:53.253202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14fd632-2f71-456e-bb5c-5d3574a7adc0 service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:53.253649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14fd632-2f71-456e-bb5c-5d3574a7adc0 service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:53.254030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14fd632-2f71-456e-bb5c-5d3574a7adc0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:53.265137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14fd632-2f71-456e-bb5c-5d3574a7adc0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:53.266970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14fd632-2f71-456e-bb5c-5d3574a7adc0 service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:53.273097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7824846c-a50a-4384-bd4f-03904af5683f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:12:53.279130 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:12:53.279929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1204/2435] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:12:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:12:53.280205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:12:53.280525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:12:53.280793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] PATH_INFO: `/v3/projects/4c65da84ed2443fbbeae23f12915eb0c` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:12:53.282317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c65da84ed2443fbbeae23f12915eb0c)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:12:53.284351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:12:53.360346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c65da84ed2443fbbeae23f12915eb0c', 'request_id': 'req-0d20b1e1-3186-45e0-89a5-637f5c6158d6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:53.366843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c65da84ed2443fbbeae23f12915eb0c', 'request_id': 'req-0d20b1e1-3186-45e0-89a5-637f5c6158d6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:53.377534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c65da84ed2443fbbeae23f12915eb0c', 'request_id': 'req-0d20b1e1-3186-45e0-89a5-637f5c6158d6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:53.392155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] The token cache is being invalidate because project 4c65da84ed2443fbbeae23f12915eb0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:12:53.392730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:53.393552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:12:53.396104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1232/2436] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:12:53 2022] DELETE /identity/v3/projects/4c65da84ed2443fbbeae23f12915eb0c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:03.275133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=79,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:03.389101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=69,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:03.680633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:03.680954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:03.681172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:03.703696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] MFA Rules not processed for user `248946c73cb245f9b8e6fe73e46133cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:03.704998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:03.805460 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1205/2437] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:13.791880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=25 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:17.982398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:17.983945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:18.044271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:18.045702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:18.105188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d20b1e1-3186-45e0-89a5-637f5c6158d6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:18.110598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:18.111252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:18.111639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:18.111908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:18.113784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:18.126627 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:18.128247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:18.139060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1233/2438] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:17 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:18.143874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7b0048-844f-48a9-b80b-8ec829f44916 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:18.148984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34483b5-5638-487c-94d2-0a1a15172167 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:18.149743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34483b5-5638-487c-94d2-0a1a15172167 service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:18.150093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34483b5-5638-487c-94d2-0a1a15172167 service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:18.150372 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34483b5-5638-487c-94d2-0a1a15172167 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:18.159403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34483b5-5638-487c-94d2-0a1a15172167 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:18.161254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34483b5-5638-487c-94d2-0a1a15172167 service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:18.174324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1206/2439] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:18 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:13:23.958356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:23.960876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:24.043791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910f67c7-f286-4c18-8c2b-137ce90775b7 service placement] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:24.050022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:24.051066 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:24.051452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:24.052127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] PATH_INFO: `/v3/users/df9113517b284ee1911b73b108dcd5bc` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:24.053278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=df9113517b284ee1911b73b108dcd5bc)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:24.055342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:24.122776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df9113517b284ee1911b73b108dcd5bc'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.123544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df9113517b284ee1911b73b108dcd5bc'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.137206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df9113517b284ee1911b73b108dcd5bc'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.138354 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df9113517b284ee1911b73b108dcd5bc'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.149676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df9113517b284ee1911b73b108dcd5bc'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.161113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:24.176884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1234/2440] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:23 2022] DELETE /identity/v3/users/df9113517b284ee1911b73b108dcd5bc => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:24.182247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:24.182959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:24.182959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:24.203205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:24.204344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:24.321731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1207/2441] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:24.335990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:24.337547 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:24.405034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f857973-98e0-4513-b551-6d98e3a98b51 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:24.411263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db2343a-f698-45af-a535-b070efb80c9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:24.411949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2343a-f698-45af-a535-b070efb80c9f service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:24.412232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2343a-f698-45af-a535-b070efb80c9f service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:24.412565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2343a-f698-45af-a535-b070efb80c9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:24.422665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db2343a-f698-45af-a535-b070efb80c9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:24.424106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db2343a-f698-45af-a535-b070efb80c9f service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:24.445067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1235/2442] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:13:24.779662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:24.781397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:24.827053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4847012-1716-4579-89e8-b194c8f4d794 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:24.833849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:24.835066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:24.835388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:24.835670 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] PATH_INFO: `/v3/projects/ae4ab41d077c49d1b7181d7b78618a6f` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:24.837048 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae4ab41d077c49d1b7181d7b78618a6f)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:24.839179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:24.955666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae4ab41d077c49d1b7181d7b78618a6f', 'request_id': 'req-00b00650-1434-4d47-9e27-0ce6515e01e0'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.966167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae4ab41d077c49d1b7181d7b78618a6f', 'request_id': 'req-00b00650-1434-4d47-9e27-0ce6515e01e0'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.976046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae4ab41d077c49d1b7181d7b78618a6f', 'request_id': 'req-00b00650-1434-4d47-9e27-0ce6515e01e0'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.997094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] The token cache is being invalidate because project ae4ab41d077c49d1b7181d7b78618a6f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:24.997386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:24.998992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-00b00650-1434-4d47-9e27-0ce6515e01e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:25.004450 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1208/2443] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:24 2022] DELETE /identity/v3/projects/ae4ab41d077c49d1b7181d7b78618a6f => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:25.007944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35922d7-9893-4ebb-9540-5baf157fbb23 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.008513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35922d7-9893-4ebb-9540-5baf157fbb23 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.008615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35922d7-9893-4ebb-9540-5baf157fbb23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.041916 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c35922d7-9893-4ebb-9540-5baf157fbb23 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:25.043223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c35922d7-9893-4ebb-9540-5baf157fbb23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:25.128512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1236/2444] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:25.136376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e503f5b-7fc0-4a79-9d29-b44a91a35b6a None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.136697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e503f5b-7fc0-4a79-9d29-b44a91a35b6a None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.136986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e503f5b-7fc0-4a79-9d29-b44a91a35b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.158612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9e503f5b-7fc0-4a79-9d29-b44a91a35b6a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:25.160080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9e503f5b-7fc0-4a79-9d29-b44a91a35b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:25.169395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1209/2445] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:25.173691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35922d7-9893-4ebb-9540-5baf157fbb23 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.188799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35922d7-9893-4ebb-9540-5baf157fbb23 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.208270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.212101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.212607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.215111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.215933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:13:25.216273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.220155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.241655 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e503f5b-7fc0-4a79-9d29-b44a91a35b6a None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.248581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1237/2446] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:13:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:13:25.252005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e503f5b-7fc0-4a79-9d29-b44a91a35b6a None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.256996 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.257682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.257971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.258430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.259288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.261420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.265177 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:25.295647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1210/2447] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:13:25 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:25.299739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.315855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdfe0ee-40e5-4246-a3b5-975652c80a88 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.393975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.394788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.395167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.395508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.399281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.399281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.419054 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:25.430888 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.432376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1238/2448] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:13:25 2022] POST /identity/v3/users => generated 292 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:13:25.446619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83413f47-ac2c-4d2d-b35e-77f2f6f57410 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.455688 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.456375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.456693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.456968 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.457788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.460159 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.471206 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1211/2449] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:13:25.475606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.486682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2a0c01-7c1f-4de0-97e2-0630668f0624 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.493969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd785d0a-77aa-413b-a4ef-9c04d24ef1ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.494644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd785d0a-77aa-413b-a4ef-9c04d24ef1ce admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.494926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd785d0a-77aa-413b-a4ef-9c04d24ef1ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.495199 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd785d0a-77aa-413b-a4ef-9c04d24ef1ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.496024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd785d0a-77aa-413b-a4ef-9c04d24ef1ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.497696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd785d0a-77aa-413b-a4ef-9c04d24ef1ce admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.511937 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1239/2450] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:25.513127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.526171 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06651b2-f8bc-493a-8e20-dd10d0995eaf admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.531429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.532346 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.532664 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.532941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] PATH_INFO: `/v3/projects/828e55688d7349c493440aa539cf59d2/users/31aa552b8e504f51afe67567039e14cd/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.557775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=828e55688d7349c493440aa539cf59d2, user_id=31aa552b8e504f51afe67567039e14cd, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.560707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.586052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1212/2451] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:25 2022] PUT /identity/v3/projects/828e55688d7349c493440aa539cf59d2/users/31aa552b8e504f51afe67567039e14cd/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:25.589203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78221f2-2d46-40fe-8389-f4cc590b011f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.589503 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78221f2-2d46-40fe-8389-f4cc590b011f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.589791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78221f2-2d46-40fe-8389-f4cc590b011f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.617910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a78221f2-2d46-40fe-8389-f4cc590b011f None None] MFA Rules not processed for user `31aa552b8e504f51afe67567039e14cd`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:25.619148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a78221f2-2d46-40fe-8389-f4cc590b011f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:25.692446 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1240/2452] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:25.697702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.777054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7aab57-fd12-4cdc-a047-7a3694a0ab46 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.781809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.782474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.782763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.783040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.783885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.785943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.812434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1213/2453] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:13:25 2022] POST /identity/v3/projects => generated 385 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:25.816429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78221f2-2d46-40fe-8389-f4cc590b011f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.828158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78221f2-2d46-40fe-8389-f4cc590b011f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.886884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.887454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.887707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.887986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.888703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.890662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.911703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1241/2454] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:13:25 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:25.915855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.931014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d3c847-8a4a-48fa-9d09-47f0175fcf97 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.935696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.936412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.936710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.936988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.937717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.939309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.950562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1214/2455] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:25.954412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.965020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd6f1ac-8f97-45af-ac02-59d344fc3e39 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:25.969841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47a3808-7d7f-4532-b5ae-1d6618786560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:25.970511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47a3808-7d7f-4532-b5ae-1d6618786560 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:25.970798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47a3808-7d7f-4532-b5ae-1d6618786560 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:25.971078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47a3808-7d7f-4532-b5ae-1d6618786560 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:25.971876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47a3808-7d7f-4532-b5ae-1d6618786560 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:25.973445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47a3808-7d7f-4532-b5ae-1d6618786560 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:25.983934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1242/2456] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:25.987958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:25.998587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c00cf-bdf8-4a23-91d2-6fd7efb6b7ba admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.003227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.004104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.004382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.004682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] PATH_INFO: `/v3/projects/047cfe67961c4e92b3ecedfa92db55da/users/cb642a4caa5249239bea949bf3b46cc4/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.021059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] RBAC: Authorizing `identity:create_grant(project_id=047cfe67961c4e92b3ecedfa92db55da, user_id=cb642a4caa5249239bea949bf3b46cc4, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.023729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.036072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1215/2457] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:25 2022] PUT /identity/v3/projects/047cfe67961c4e92b3ecedfa92db55da/users/cb642a4caa5249239bea949bf3b46cc4/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:26.046396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec9a89-4f33-43c2-984d-3055087baf16 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.046880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec9a89-4f33-43c2-984d-3055087baf16 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.046975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec9a89-4f33-43c2-984d-3055087baf16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.073729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-72ec9a89-4f33-43c2-984d-3055087baf16 None None] MFA Rules not processed for user `cb642a4caa5249239bea949bf3b46cc4`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:26.074939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-72ec9a89-4f33-43c2-984d-3055087baf16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:26.119508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:26.140557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1243/2458] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:26 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:26.145111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:26.206969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3c1ff5-f88d-4d86-91a4-e2d6901ae381 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.212042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.212811 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.213093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.213368 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.214066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.216032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.223681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:26.254380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1216/2459] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:13:26 2022] POST /identity/v3/projects => generated 383 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:26.255479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ec9a89-4f33-43c2-984d-3055087baf16 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:26.270008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ec9a89-4f33-43c2-984d-3055087baf16 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.335444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.336216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.336608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.336955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.337855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.339834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.364008 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1244/2460] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:13:26 2022] POST /identity/v3/users => generated 298 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:13:26.365045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:26.375986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35104f10-5be2-4b99-8c9c-a6f9caa1c7df admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.380694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.381408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.381684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.381961 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.382723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.384309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.400312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1217/2461] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:13:26.404164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:26.415445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9760cf45-1250-42e3-ab34-98cd63906fda admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.422632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.423655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.423941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.424217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] PATH_INFO: `/v3/projects/b44ee3f02d624d8096956435031fe15a/users/24932a4ed8c64b4fb7dbb69a1a1dcdc8/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.451758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=b44ee3f02d624d8096956435031fe15a, user_id=24932a4ed8c64b4fb7dbb69a1a1dcdc8, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.454521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.467091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1245/2462] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:26 2022] PUT /identity/v3/projects/b44ee3f02d624d8096956435031fe15a/users/24932a4ed8c64b4fb7dbb69a1a1dcdc8/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:13:26.470943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:26.482377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091a6696-c90c-4b55-a33f-e5279be1154a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.542143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59de90dd-7918-466b-965d-b7ee84bcd7cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.542817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59de90dd-7918-466b-965d-b7ee84bcd7cb admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.543104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59de90dd-7918-466b-965d-b7ee84bcd7cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.543380 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59de90dd-7918-466b-965d-b7ee84bcd7cb admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.544102 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59de90dd-7918-466b-965d-b7ee84bcd7cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.545723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59de90dd-7918-466b-965d-b7ee84bcd7cb admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.556265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1218/2463] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:26 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:26.560236 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:26.570366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e2c451-9b2c-4ce1-8dde-2be49c7076dd admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.575033 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.575904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.576180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.576481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] PATH_INFO: `/v3/projects/b44ee3f02d624d8096956435031fe15a/users/24932a4ed8c64b4fb7dbb69a1a1dcdc8/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.578412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b44ee3f02d624d8096956435031fe15a, user_id=24932a4ed8c64b4fb7dbb69a1a1dcdc8, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.581083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.591255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1246/2464] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:26 2022] PUT /identity/v3/projects/b44ee3f02d624d8096956435031fe15a/users/24932a4ed8c64b4fb7dbb69a1a1dcdc8/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:13:26.602306 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2353a6c7-f25c-4211-989e-e0a1700b948b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.602598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2353a6c7-f25c-4211-989e-e0a1700b948b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.602874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2353a6c7-f25c-4211-989e-e0a1700b948b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.628871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-2353a6c7-f25c-4211-989e-e0a1700b948b None None] MFA Rules not processed for user `24932a4ed8c64b4fb7dbb69a1a1dcdc8`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:26.630075 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2353a6c7-f25c-4211-989e-e0a1700b948b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:26.696993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1219/2465] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:26 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:26.708012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:26.710553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:26.789229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24da51f0-75c5-4ef9-859e-14748c4a8046 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:26.792883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:26.793528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:26.793815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:26.794095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:26.828748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:26.856126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:26.857602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:26.866377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1247/2466] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:27.640401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca10bd-1b66-4292-af46-2686c08a559f None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:27.640876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca10bd-1b66-4292-af46-2686c08a559f None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:27.641017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca10bd-1b66-4292-af46-2686c08a559f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:27.668055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-7dca10bd-1b66-4292-af46-2686c08a559f None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:27.669228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7dca10bd-1b66-4292-af46-2686c08a559f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:27.711588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:27.731654 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1220/2467] 10.222.0.20 () {58 vars in 967 bytes} [Tue Jul 5 10:13:27 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:27.746338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:27.747762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:27.816872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23946405-51a5-453e-9ebc-d06be826930a service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:27.821754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:27.822396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:27.822677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:27.822954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:27.830948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:27.832569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:27.840787 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1248/2468] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:13:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:13:29.408959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca10bd-1b66-4292-af46-2686c08a559f None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:29.421447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca10bd-1b66-4292-af46-2686c08a559f None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:29.425410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:29.426164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:29.426538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:29.426815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:29.498223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:29.500127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:29.511636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1221/2469] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:30.083002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:30.084847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:30.171045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:30.191411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333968f2-0a5b-4692-af39-16604eddef4c service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:30.195331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:30.196001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.196289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.196603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.207927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:30.208510 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9669fa8-891a-45be-becc-038efd0e5b41 service swift] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:30.209406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:30.217377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:30.218590 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.219109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.219658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] PATH_INFO: `/v3/users/d2d25be9c4674aa09b2975f1c6bc8c52` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.221235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2d25be9c4674aa09b2975f1c6bc8c52)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:30.224727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:30.230977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1222/2470] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:30 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:30.284508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2d25be9c4674aa09b2975f1c6bc8c52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.285365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2d25be9c4674aa09b2975f1c6bc8c52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.294340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2d25be9c4674aa09b2975f1c6bc8c52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.295021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2d25be9c4674aa09b2975f1c6bc8c52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.304243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-914a0fab-fbf8-41c5-9eb5-1a54140118d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2d25be9c4674aa09b2975f1c6bc8c52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.306773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:30.328471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1249/2471] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:30 2022] DELETE /identity/v3/users/d2d25be9c4674aa09b2975f1c6bc8c52 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:30.502089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:30.513012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f74e6e6-fa6f-4cf9-a130-f07044405fd6 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:30.575868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:30.576773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.577049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.577326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] PATH_INFO: `/v3/projects/c48f677354594fdc839367fff8b3dc29` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.578643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] RBAC: Authorizing `identity:delete_project(project_id=c48f677354594fdc839367fff8b3dc29)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:30.580983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:30.634941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c48f677354594fdc839367fff8b3dc29', 'request_id': 'req-4e0255c9-57ce-47b6-b1d2-279c89bf438f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.640825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c48f677354594fdc839367fff8b3dc29', 'request_id': 'req-4e0255c9-57ce-47b6-b1d2-279c89bf438f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.647300 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c48f677354594fdc839367fff8b3dc29', 'request_id': 'req-4e0255c9-57ce-47b6-b1d2-279c89bf438f'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.657630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:30.662362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] The token cache is being invalidate because project c48f677354594fdc839367fff8b3dc29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:30.662733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.664147 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4e0255c9-57ce-47b6-b1d2-279c89bf438f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:30.666247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1223/2472] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:30 2022] DELETE /identity/v3/projects/c48f677354594fdc839367fff8b3dc29 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:30.672116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f858f00-71be-4f6a-89f6-5d9a5b62cae7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.672412 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f858f00-71be-4f6a-89f6-5d9a5b62cae7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.672699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f858f00-71be-4f6a-89f6-5d9a5b62cae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.690700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9f858f00-71be-4f6a-89f6-5d9a5b62cae7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:30.691882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9f858f00-71be-4f6a-89f6-5d9a5b62cae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:30.752530 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1250/2473] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:30.759908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d5e00e-cea6-4b0a-b3d9-2c01dd5cf752 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.760278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d5e00e-cea6-4b0a-b3d9-2c01dd5cf752 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.760653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d5e00e-cea6-4b0a-b3d9-2c01dd5cf752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.779172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d1d5e00e-cea6-4b0a-b3d9-2c01dd5cf752 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:30.780420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d1d5e00e-cea6-4b0a-b3d9-2c01dd5cf752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:30.788851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1224/2474] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:30.792951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f858f00-71be-4f6a-89f6-5d9a5b62cae7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:30.803152 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f858f00-71be-4f6a-89f6-5d9a5b62cae7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:30.807634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:30.808275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.808591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.808871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.809589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:13:30.809890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:30.811479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:30.823341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1251/2475] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:13:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:13:30.830125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d5e00e-cea6-4b0a-b3d9-2c01dd5cf752 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:30.840323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d5e00e-cea6-4b0a-b3d9-2c01dd5cf752 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:30.845006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:30.845657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.845941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.846222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.846904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:30.848848 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:30.877850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1225/2476] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:13:30 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:30.878986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:30.889753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c2d389-41f9-4c92-850f-5f4d2ce0b9c4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:30.951646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:30.952325 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.952636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.952913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:30.955569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:30.976246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1252/2477] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:13:30 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:30.980607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:30.991331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee963509-26a7-4c81-a5f7-70914b9d329f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:30.995930 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:30.996636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:30.996919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:30.997224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:30.997977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:30.999572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:31.011537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1226/2478] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:31.015553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:31.026957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dce9e0-bd63-4548-8369-cc71384ae7ef admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:31.031660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9655068-1cd3-4513-90c9-804d174ca563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:31.032373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9655068-1cd3-4513-90c9-804d174ca563 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:31.032695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9655068-1cd3-4513-90c9-804d174ca563 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:31.032976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9655068-1cd3-4513-90c9-804d174ca563 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:31.033699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9655068-1cd3-4513-90c9-804d174ca563 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:31.035309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9655068-1cd3-4513-90c9-804d174ca563 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:31.045213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1253/2479] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:31.049283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:31.059058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a38262-e4e5-48b1-a163-dc31251ab9bd admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:31.063572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:31.064479 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:31.064763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:31.065046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] PATH_INFO: `/v3/projects/accabbba722345d190b52446d1e7c2b7/users/df495d1fbab24736a2ba12e8a0e7d803/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:31.079501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] RBAC: Authorizing `identity:create_grant(project_id=accabbba722345d190b52446d1e7c2b7, user_id=df495d1fbab24736a2ba12e8a0e7d803, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:31.082180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:31.095574 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1227/2480] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:31 2022] PUT /identity/v3/projects/accabbba722345d190b52446d1e7c2b7/users/df495d1fbab24736a2ba12e8a0e7d803/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:31.100555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dbe8b6-c0e2-4691-98fe-d73c22a92f65 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:31.101000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dbe8b6-c0e2-4691-98fe-d73c22a92f65 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:31.101109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dbe8b6-c0e2-4691-98fe-d73c22a92f65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:31.118692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c9dbe8b6-c0e2-4691-98fe-d73c22a92f65 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:31.119881 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c9dbe8b6-c0e2-4691-98fe-d73c22a92f65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:31.164434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:31.181993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1254/2481] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:31.191921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:31.193709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:31.259049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7143f6-c91f-40ba-bf62-91336376a184 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:31.263493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:31.264223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:31.264603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:31.264965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:31.276309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:31.277780 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:31.290058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1228/2482] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:32.052764 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2620d8f2-0ba9-4066-b153-1e8559cc10d0 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:32.053314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2620d8f2-0ba9-4066-b153-1e8559cc10d0 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:32.054187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2620d8f2-0ba9-4066-b153-1e8559cc10d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:32.082523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2620d8f2-0ba9-4066-b153-1e8559cc10d0 None None] MFA Rules not processed for user `df495d1fbab24736a2ba12e8a0e7d803`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:32.083857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2620d8f2-0ba9-4066-b153-1e8559cc10d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:32.150632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1255/2483] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:32.161485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:32.162920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:32.220656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185090db-11fb-4516-9f6f-93f9305852eb service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:32.224226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:32.224915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:32.225199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:32.225475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:32.234643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:32.236203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:32.245955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1229/2484] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:34.651825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a5383-48bc-48cc-9d81-ebb36d1ee365 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:34.652313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a5383-48bc-48cc-9d81-ebb36d1ee365 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:34.652313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a5383-48bc-48cc-9d81-ebb36d1ee365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:34.669429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5e3a5383-48bc-48cc-9d81-ebb36d1ee365 None None] MFA Rules not processed for user `df495d1fbab24736a2ba12e8a0e7d803`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:34.670606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5e3a5383-48bc-48cc-9d81-ebb36d1ee365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:34.677389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1256/2485] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:35.180462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:35.181916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:35.197643 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09227d7-2cbd-42ae-9632-fda74cd34f60 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:35.202355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:35.203331 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:35.203627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:35.203979 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] PATH_INFO: `/v3/users/31aa552b8e504f51afe67567039e14cd` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:35.205575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31aa552b8e504f51afe67567039e14cd)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:35.207458 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:35.251710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:35.263823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31aa552b8e504f51afe67567039e14cd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:35.276258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31aa552b8e504f51afe67567039e14cd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:35.277123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31aa552b8e504f51afe67567039e14cd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:35.277816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31aa552b8e504f51afe67567039e14cd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:35.293769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31aa552b8e504f51afe67567039e14cd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:35.311602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1230/2486] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:35 2022] DELETE /identity/v3/users/31aa552b8e504f51afe67567039e14cd => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:35.317318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:35.317629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:35.317935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:35.341068 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:35.342587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:35.423141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1257/2487] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:35.430103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:35.431612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:35.496770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f369c7aa-268a-41ed-b68b-4c5a1a00b8a3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:35.500572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:35.501236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:35.501524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:35.501809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:35.512134 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:35.513753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:35.525695 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1231/2488] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:35.873810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:35.875268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:35.895034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8954ab6-3f06-47c4-b554-87b6038e4a58 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:35.909937 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:35.910822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:35.911104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:35.911380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] PATH_INFO: `/v3/projects/828e55688d7349c493440aa539cf59d2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:35.914252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] RBAC: Authorizing `identity:delete_project(project_id=828e55688d7349c493440aa539cf59d2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:35.916837 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:35.988478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '828e55688d7349c493440aa539cf59d2', 'request_id': 'req-7f558f49-caf5-4c18-ba8b-9d1ebc668505'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:35.996038 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '828e55688d7349c493440aa539cf59d2', 'request_id': 'req-7f558f49-caf5-4c18-ba8b-9d1ebc668505'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.007857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '828e55688d7349c493440aa539cf59d2', 'request_id': 'req-7f558f49-caf5-4c18-ba8b-9d1ebc668505'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.011107 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:36.028319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] The token cache is being invalidate because project 828e55688d7349c493440aa539cf59d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:36.029271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.029942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.035861 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1258/2489] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:35 2022] DELETE /identity/v3/projects/828e55688d7349c493440aa539cf59d2 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:36.037609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:36.052803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb32db-0a90-48d1-b19d-541dc3b239f9 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:36.135610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:36.136731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:36.137062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:36.137370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] PATH_INFO: `/v3/users/cb642a4caa5249239bea949bf3b46cc4` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:36.139024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb642a4caa5249239bea949bf3b46cc4)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:36.141277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:36.200112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb642a4caa5249239bea949bf3b46cc4'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.209291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb642a4caa5249239bea949bf3b46cc4'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.210796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb642a4caa5249239bea949bf3b46cc4'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.211552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb642a4caa5249239bea949bf3b46cc4'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.215202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:36.238742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb642a4caa5249239bea949bf3b46cc4'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.259145 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1232/2490] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:36 2022] DELETE /identity/v3/users/cb642a4caa5249239bea949bf3b46cc4 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:36.574505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:36.576060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:36.596751 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f558f49-caf5-4c18-ba8b-9d1ebc668505 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:36.659620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:36.661407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:36.661793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:36.662173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] PATH_INFO: `/v3/projects/047cfe67961c4e92b3ecedfa92db55da` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:36.664347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=047cfe67961c4e92b3ecedfa92db55da)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:36.666378 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:36.749517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '047cfe67961c4e92b3ecedfa92db55da', 'request_id': 'req-8f10979e-399a-4594-816e-069668ee4bd0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.757331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '047cfe67961c4e92b3ecedfa92db55da', 'request_id': 'req-8f10979e-399a-4594-816e-069668ee4bd0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.770785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '047cfe67961c4e92b3ecedfa92db55da', 'request_id': 'req-8f10979e-399a-4594-816e-069668ee4bd0'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.802888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] The token cache is being invalidate because project 047cfe67961c4e92b3ecedfa92db55da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:36.803286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.804238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:36.810320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1259/2491] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:36 2022] DELETE /identity/v3/projects/047cfe67961c4e92b3ecedfa92db55da => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:36.811063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:36.812550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:36.827987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec955af3-802e-4c31-a6b7-d8d4128d89ca admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:36.932633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:36.932633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:36.932633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:36.938361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] PATH_INFO: `/v3/users/24932a4ed8c64b4fb7dbb69a1a1dcdc8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:36.939782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=24932a4ed8c64b4fb7dbb69a1a1dcdc8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:36.941934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:37.023967 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24932a4ed8c64b4fb7dbb69a1a1dcdc8'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.037481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24932a4ed8c64b4fb7dbb69a1a1dcdc8'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.039131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24932a4ed8c64b4fb7dbb69a1a1dcdc8'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24932a4ed8c64b4fb7dbb69a1a1dcdc8'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.060235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-3aa70581-fcfa-4a28-be97-f353e2a06e55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24932a4ed8c64b4fb7dbb69a1a1dcdc8'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.088437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1233/2492] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:36 2022] DELETE /identity/v3/users/24932a4ed8c64b4fb7dbb69a1a1dcdc8 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:37.498852 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:37.501435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:37.516317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f10979e-399a-4594-816e-069668ee4bd0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:37.602616 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:37.603522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:37.603812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:37.604144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] PATH_INFO: `/v3/projects/b44ee3f02d624d8096956435031fe15a` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:37.605464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b44ee3f02d624d8096956435031fe15a)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:37.607411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:37.638621 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:37.655472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b44ee3f02d624d8096956435031fe15a', 'request_id': 'req-372390e3-70c2-4ef5-9ac2-a0d3797fd037'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.660939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b44ee3f02d624d8096956435031fe15a', 'request_id': 'req-372390e3-70c2-4ef5-9ac2-a0d3797fd037'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.668058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b44ee3f02d624d8096956435031fe15a', 'request_id': 'req-372390e3-70c2-4ef5-9ac2-a0d3797fd037'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.679711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] The token cache is being invalidate because project b44ee3f02d624d8096956435031fe15a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:37.680000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.680679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-372390e3-70c2-4ef5-9ac2-a0d3797fd037 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:37.682233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1260/2493] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:37 2022] DELETE /identity/v3/projects/b44ee3f02d624d8096956435031fe15a => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:37.687037 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8393f69-8300-4089-a7b7-33c0e8ee884c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:37.687512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8393f69-8300-4089-a7b7-33c0e8ee884c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:37.687637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8393f69-8300-4089-a7b7-33c0e8ee884c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:37.716669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f8393f69-8300-4089-a7b7-33c0e8ee884c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:37.718141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f8393f69-8300-4089-a7b7-33c0e8ee884c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:37.767592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:37.783865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1234/2494] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:37.792009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f9fa7-0b7f-4fca-a98f-59b0db2996f7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:37.792302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f9fa7-0b7f-4fca-a98f-59b0db2996f7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:37.792612 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f9fa7-0b7f-4fca-a98f-59b0db2996f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:37.812018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5a3f9fa7-0b7f-4fca-a98f-59b0db2996f7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:37.813367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5a3f9fa7-0b7f-4fca-a98f-59b0db2996f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:37.821517 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1261/2495] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:37.830861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8393f69-8300-4089-a7b7-33c0e8ee884c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:37.841110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8393f69-8300-4089-a7b7-33c0e8ee884c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:37.845801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:37.846467 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:37.846748 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:37.847021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:37.847742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:13:37.848049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:37.849679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:37.867052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1235/2496] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:13:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:13:37.871288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f9fa7-0b7f-4fca-a98f-59b0db2996f7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:37.883076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f9fa7-0b7f-4fca-a98f-59b0db2996f7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:37.885791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:37.886445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:37.886730 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:37.887004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:37.887701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:37.889630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:37.920292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1262/2497] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:13:37 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:37.921680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:37.932862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f23242-7dbf-4999-abd0-7aa730cdc3ef admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:38.003469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:38.004281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:38.004739 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:38.005012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:38.005718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:38.007706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:38.033454 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1236/2498] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:13:37 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:13:38.037360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:38.060831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66f558a-64a9-4dbb-a113-71ec334ef4e4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:38.067290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:38.068111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:38.068507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:38.068867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:38.069919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:38.071845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:38.087531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1263/2499] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:38 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:13:38.091384 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:38.106185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672df6aa-b81e-4469-a377-645d3d4e2eb8 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:38.117302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c93234e-557d-4e29-bc29-d0d857468bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:38.119504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c93234e-557d-4e29-bc29-d0d857468bc6 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:38.119935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c93234e-557d-4e29-bc29-d0d857468bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:38.120404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c93234e-557d-4e29-bc29-d0d857468bc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:38.121696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c93234e-557d-4e29-bc29-d0d857468bc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:38.126704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c93234e-557d-4e29-bc29-d0d857468bc6 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:38.146867 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1237/2500] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:38 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:38.149763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:38.161540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b24870-c8a6-4bec-8785-c858ce7ab06f admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:38.167580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:38.168690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:38.169025 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:38.169376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] PATH_INFO: `/v3/projects/ee432e64b3c84f3b83227bfa25e4293b/users/3aee95220e114ab3ab48b74873d705b6/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:38.189780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee432e64b3c84f3b83227bfa25e4293b, user_id=3aee95220e114ab3ab48b74873d705b6, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:38.194064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:38.228137 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1264/2501] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:38 2022] PUT /identity/v3/projects/ee432e64b3c84f3b83227bfa25e4293b/users/3aee95220e114ab3ab48b74873d705b6/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:38.239164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b06ffb-5045-47a3-a30e-62c4ed4d0e57 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:38.239465 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b06ffb-5045-47a3-a30e-62c4ed4d0e57 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:38.239799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b06ffb-5045-47a3-a30e-62c4ed4d0e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:38.281163 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e7b06ffb-5045-47a3-a30e-62c4ed4d0e57 None None] MFA Rules not processed for user `3aee95220e114ab3ab48b74873d705b6`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:38.282909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e7b06ffb-5045-47a3-a30e-62c4ed4d0e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:38.361551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1238/2502] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:38 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:38.377560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:38.379296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:38.437086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22b5b9c-d2de-484c-aa89-8118f7a35d30 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:38.440876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:38.441626 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:38.441986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:38.442263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:38.451297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:38.452836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:38.462679 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1265/2503] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:38 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:48.174155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:48.174598 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:48.174713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:48.238807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:48.240047 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:48.278297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:48.364977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1239/2504] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:13:48 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:48.386080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:48.387640 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:48.388945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:48.390310 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:48.472628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:48.482850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f366580-7f36-475b-adbb-796ed86dccb2 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:48.499700 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:48.500685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:48.500971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:48.501250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] PATH_INFO: `/v3/users/2d94be4d09b04c6b90c9fcfb6685546c` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:48.502597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d94be4d09b04c6b90c9fcfb6685546c)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:48.504602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:48.530260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d678e5f2-54d2-458a-982d-a4b67be28ff3 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:48.548552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9def4a-2f4d-4002-b4c7-a5d384973ea6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:48.549394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9def4a-2f4d-4002-b4c7-a5d384973ea6 service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:48.550335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9def4a-2f4d-4002-b4c7-a5d384973ea6 service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:48.550725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9def4a-2f4d-4002-b4c7-a5d384973ea6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:48.562657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d94be4d09b04c6b90c9fcfb6685546c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:48.563346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d94be4d09b04c6b90c9fcfb6685546c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:48.564889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9def4a-2f4d-4002-b4c7-a5d384973ea6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:48.566964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9def4a-2f4d-4002-b4c7-a5d384973ea6 service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:48.577957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1240/2505] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:13:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:13:48.591884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d94be4d09b04c6b90c9fcfb6685546c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:48.594024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d94be4d09b04c6b90c9fcfb6685546c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:48.608856 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d94be4d09b04c6b90c9fcfb6685546c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:48.638838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1266/2506] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:48 2022] DELETE /identity/v3/users/2d94be4d09b04c6b90c9fcfb6685546c => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:48.643796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e022e-3d97-4d4e-85c5-0438545325ac None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:48.644093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e022e-3d97-4d4e-85c5-0438545325ac None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:48.644382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e022e-3d97-4d4e-85c5-0438545325ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:48.668579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5a2e022e-3d97-4d4e-85c5-0438545325ac None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:48.669845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5a2e022e-3d97-4d4e-85c5-0438545325ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:48.809787 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1241/2507] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:48.815155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:48.816849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:48.917814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720d9997-0754-4a2b-98c5-6519805898a2 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:48.953097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:48.954644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:48.955567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:48.956533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:48.989285 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:48.990747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:49.012364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1267/2508] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:13:49.352034 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:49.352571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:49.352777 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:49.395668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:49.397021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:49.405112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:49.407634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:49.428720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdc5a-16a1-43a3-aff2-de214f158a85 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:49.443052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:49.445842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:49.446150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:49.446435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] PATH_INFO: `/v3/projects/9a96afb5d6054ddcbf3b4cf54a2900b7` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:49.447943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a96afb5d6054ddcbf3b4cf54a2900b7)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:49.449995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:49.471594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:49.512662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1242/2509] 10.222.0.20 () {58 vars in 967 bytes} [Tue Jul 5 10:13:49 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:49.530818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:49.532755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:49.542144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a96afb5d6054ddcbf3b4cf54a2900b7', 'request_id': 'req-3f50600c-796c-4846-bf65-31c78dae98f1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:49.565250 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a96afb5d6054ddcbf3b4cf54a2900b7', 'request_id': 'req-3f50600c-796c-4846-bf65-31c78dae98f1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:49.581635 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a96afb5d6054ddcbf3b4cf54a2900b7', 'request_id': 'req-3f50600c-796c-4846-bf65-31c78dae98f1'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:49.595381 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:49.600131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] The token cache is being invalidate because project 9a96afb5d6054ddcbf3b4cf54a2900b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:49.601162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:49.602778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f50600c-796c-4846-bf65-31c78dae98f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:49.610982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1268/2510] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:49 2022] DELETE /identity/v3/projects/9a96afb5d6054ddcbf3b4cf54a2900b7 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:49.612880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb93b4-79b8-42a7-969e-7af22f524da1 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:49.614167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb93b4-79b8-42a7-969e-7af22f524da1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:49.614646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb93b4-79b8-42a7-969e-7af22f524da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:49.645846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ccdb93b4-79b8-42a7-969e-7af22f524da1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:49.647257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ccdb93b4-79b8-42a7-969e-7af22f524da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:49.678907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6303bcb0-6d92-4327-bbb5-ebebd5372db4 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:49.763001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1269/2511] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:49.770136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:49.771642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e098ac-0e1f-4f94-bdb0-2004f2d645ad None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:49.771642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e098ac-0e1f-4f94-bdb0-2004f2d645ad None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:49.771642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e098ac-0e1f-4f94-bdb0-2004f2d645ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:49.772186 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:49.772661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:49.773090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:49.776431 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:49.793631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-21e098ac-0e1f-4f94-bdb0-2004f2d645ad None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:49.794846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-21e098ac-0e1f-4f94-bdb0-2004f2d645ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:49.803344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1270/2512] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:13:49.810643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e098ac-0e1f-4f94-bdb0-2004f2d645ad None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:49.821519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e098ac-0e1f-4f94-bdb0-2004f2d645ad None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:49.826724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:49.827396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:49.827675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:49.827951 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:49.828722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:13:49.829340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:49.831139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:49.861368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1271/2513] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:13:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:13:49.862520 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:49.899663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb675950-22fd-4f03-8f1b-1ff2d07462ca admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:49.905484 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:49.906299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:49.907521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:49.907849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:49.908232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:49.908881 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:49.909156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:49.911479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:49.930711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1243/2514] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:13:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 05 10:13:49.993985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1272/2515] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:13:49 2022] POST /identity/v3/projects => generated 361 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:49.996826 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:50.008935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcdf088-0007-4610-ab4a-bc82a6459d6d service swift] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:50.031085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=100 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:50.102253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8435ff0c-4b48-4a37-bc61-bd2b7234adb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:50.103327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8435ff0c-4b48-4a37-bc61-bd2b7234adb0 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.103840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8435ff0c-4b48-4a37-bc61-bd2b7234adb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.104335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8435ff0c-4b48-4a37-bc61-bd2b7234adb0 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.105480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8435ff0c-4b48-4a37-bc61-bd2b7234adb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:50.108376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8435ff0c-4b48-4a37-bc61-bd2b7234adb0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:50.136929 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1244/2516] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:13:49 2022] POST /identity/v3/users => generated 281 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:50.141321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:50.155213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c32a5d-da78-4ab0-a1dc-01e173ae42f5 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:50.160969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:50.161651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.161964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.162243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.162975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:50.164650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:50.186184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.186482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.186771 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.187580 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1273/2517] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:50 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:50.188874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:50.199179 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf61c9-7cf1-4957-99cd-b4a12e546947 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:50.204341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:50.205039 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.205321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.205596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.206334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:50.207012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:50.208078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:50.208248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:50.224574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1274/2518] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:13:50.228379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:50.240959 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80486fbb-80a0-4b4c-ac4d-12627a49e51d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:50.250982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22cc311-3daf-4bca-8ffb-9f9b3ff6cf8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:50.252123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22cc311-3daf-4bca-8ffb-9f9b3ff6cf8a admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.252506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22cc311-3daf-4bca-8ffb-9f9b3ff6cf8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22cc311-3daf-4bca-8ffb-9f9b3ff6cf8a admin admin] PATH_INFO: `/v3/projects/e008272f22d74b0e97fdc329a3ec7763/users/572788c3d7a24fe4afe6b223801f3192/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.278083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22cc311-3daf-4bca-8ffb-9f9b3ff6cf8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e008272f22d74b0e97fdc329a3ec7763, user_id=572788c3d7a24fe4afe6b223801f3192, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:50.280816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22cc311-3daf-4bca-8ffb-9f9b3ff6cf8a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:50.282860 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1245/2519] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:13:50 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:13:50.296507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:50.297346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1275/2520] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:50 2022] PUT /identity/v3/projects/e008272f22d74b0e97fdc329a3ec7763/users/572788c3d7a24fe4afe6b223801f3192/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:50.302128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:50.309084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.309448 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.309809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.346453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] MFA Rules not processed for user `572788c3d7a24fe4afe6b223801f3192`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:50.348098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:50.413371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31be356-8aab-4ce6-a0d8-677025e48525 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:50.430375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1276/2521] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:50 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:13:50.446408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:50.449090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:50.456653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:50.505090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:50.518045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:50.518875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.520055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.520364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.567043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65fa1ff-f156-48e1-94f8-8b7366ef42f1 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:50.576089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ee7462-4c53-4ef6-ba70-2a5269dc23e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:50.576823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ee7462-4c53-4ef6-ba70-2a5269dc23e7 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:50.577080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ee7462-4c53-4ef6-ba70-2a5269dc23e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:50.577360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ee7462-4c53-4ef6-ba70-2a5269dc23e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:50.602298 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:50.602560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ee7462-4c53-4ef6-ba70-2a5269dc23e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:50.603915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ee7462-4c53-4ef6-ba70-2a5269dc23e7 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:50.604030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:50.615795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1277/2522] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:50 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:13:50.633491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1246/2523] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:13:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 05 10:13:51.161166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:51.161406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:51.161727 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:51.215226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:51.216526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:51.225376 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1278/2524] 10.222.0.20 () {58 vars in 967 bytes} [Tue Jul 5 10:13:51 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:51.246980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:51.270660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97ce8a-8c47-498a-912a-e63b5260702b service swift] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:51.285103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09ba0b1-4c28-41d7-b7a4-81b120cf1aa6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:51.287471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ba0b1-4c28-41d7-b7a4-81b120cf1aa6 service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:51.290290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ba0b1-4c28-41d7-b7a4-81b120cf1aa6 service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:51.290774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ba0b1-4c28-41d7-b7a4-81b120cf1aa6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:51.313489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09ba0b1-4c28-41d7-b7a4-81b120cf1aa6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:51.315406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09ba0b1-4c28-41d7-b7a4-81b120cf1aa6 service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:51.338597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1247/2525] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:13:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:13:53.757618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:53.759078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:53.824825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3be1e9-d519-499e-ac73-b94f9865e66f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:53.829529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:53.830394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:53.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:53.830961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] PATH_INFO: `/v3/users/aa8cee3ae343415aae94ce2592a04dd3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:53.832209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] RBAC: Authorizing `identity:delete_user(user_id=aa8cee3ae343415aae94ce2592a04dd3)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:53.834173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:53.884425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aa8cee3ae343415aae94ce2592a04dd3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:53.885112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aa8cee3ae343415aae94ce2592a04dd3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:53.892384 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aa8cee3ae343415aae94ce2592a04dd3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:53.893101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aa8cee3ae343415aae94ce2592a04dd3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:53.901965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aa8cee3ae343415aae94ce2592a04dd3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:53.922173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1279/2526] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:53 2022] DELETE /identity/v3/users/aa8cee3ae343415aae94ce2592a04dd3 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:53.926621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:53.927084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:53.927168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:53.948981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:53.958786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:54.032673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1248/2527] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:54.042658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.044101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:54.072745 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:54.122256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcabec5-ef72-42e7-92a7-b768c06184db admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.126319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d4591e-43e6-4e47-92f2-c84334dd6e2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.126976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4591e-43e6-4e47-92f2-c84334dd6e2d service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.127267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4591e-43e6-4e47-92f2-c84334dd6e2d service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.127544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4591e-43e6-4e47-92f2-c84334dd6e2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.138037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d4591e-43e6-4e47-92f2-c84334dd6e2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.139480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d4591e-43e6-4e47-92f2-c84334dd6e2d service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.152816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1280/2528] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:54.363686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.365160 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:54.395928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da4c490-ff7f-40a7-8bfd-8aa15dc1a114 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.401822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.402770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.402962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.403244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] PATH_INFO: `/v3/projects/9bf4c2e6ff4347f5b5e46efe5d431dc9` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.404971 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=9bf4c2e6ff4347f5b5e46efe5d431dc9)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.406924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.464723 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9bf4c2e6ff4347f5b5e46efe5d431dc9', 'request_id': 'req-f18ec608-61e8-459f-9cb0-724247a6b3ea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:54.470515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9bf4c2e6ff4347f5b5e46efe5d431dc9', 'request_id': 'req-f18ec608-61e8-459f-9cb0-724247a6b3ea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:54.477110 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9bf4c2e6ff4347f5b5e46efe5d431dc9', 'request_id': 'req-f18ec608-61e8-459f-9cb0-724247a6b3ea'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:54.491568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] The token cache is being invalidate because project 9bf4c2e6ff4347f5b5e46efe5d431dc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:54.492018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:54.493207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-f18ec608-61e8-459f-9cb0-724247a6b3ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:54.494763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1249/2529] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:54 2022] DELETE /identity/v3/projects/9bf4c2e6ff4347f5b5e46efe5d431dc9 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:13:54.499596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a04299c-c481-4c89-995b-c1d0c51917f7 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.499882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a04299c-c481-4c89-995b-c1d0c51917f7 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.500157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a04299c-c481-4c89-995b-c1d0c51917f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.520129 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7a04299c-c481-4c89-995b-c1d0c51917f7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:54.521278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7a04299c-c481-4c89-995b-c1d0c51917f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:54.588488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1281/2530] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:54.596719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d5fc6-2fe8-4f00-a0b0-aece0970704b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.597012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d5fc6-2fe8-4f00-a0b0-aece0970704b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.597783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3d5fc6-2fe8-4f00-a0b0-aece0970704b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.616138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1d3d5fc6-2fe8-4f00-a0b0-aece0970704b None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:54.617917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1d3d5fc6-2fe8-4f00-a0b0-aece0970704b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:54.625915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1250/2531] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:54.630065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a04299c-c481-4c89-995b-c1d0c51917f7 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.640323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a04299c-c481-4c89-995b-c1d0c51917f7 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.645117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.645783 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.646063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.646335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.647061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:13:54.647365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.649059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.661050 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1282/2532] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:13:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:13:54.665462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3d5fc6-2fe8-4f00-a0b0-aece0970704b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.671054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:54.675233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3d5fc6-2fe8-4f00-a0b0-aece0970704b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.681653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.682543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.683043 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.683349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.684117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.686301 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.712397 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1251/2533] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:13:54 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:54.716546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.726355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd074cf-82eb-49df-a874-b75e1421df8e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.783094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.783697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.783937 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.784211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.784928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.786831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.807984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1283/2534] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:13:54 2022] POST /identity/v3/users => generated 292 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:13:54.812030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.822229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9b4483-4740-4821-a83d-554b079f6f6b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.826756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.827427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.827699 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.827978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.828747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.830345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.841244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1252/2535] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:54 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:54.844578 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.855134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aab1348-f543-4d79-90be-8aff13398f18 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.859709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2530f500-a86a-4478-bcd2-02343d261553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.860359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2530f500-a86a-4478-bcd2-02343d261553 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.860678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2530f500-a86a-4478-bcd2-02343d261553 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.860957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2530f500-a86a-4478-bcd2-02343d261553 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.861675 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2530f500-a86a-4478-bcd2-02343d261553 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.863276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2530f500-a86a-4478-bcd2-02343d261553 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.873735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1284/2536] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:13:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:13:54.877666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:54.887610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7c603b-1b12-4f8a-b2d2-a58a0b85a2f0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:54.892120 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:54.893022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.893303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.893578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] PATH_INFO: `/v3/projects/d71ebc02c6094919b3054d7da84aadf0/users/f0351b9ab5ac4fbaad38df073518412a/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.907759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d71ebc02c6094919b3054d7da84aadf0, user_id=f0351b9ab5ac4fbaad38df073518412a, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:54.910434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:54.924253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1253/2537] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:13:54 2022] PUT /identity/v3/projects/d71ebc02c6094919b3054d7da84aadf0/users/f0351b9ab5ac4fbaad38df073518412a/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:13:54.930291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:54.930752 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:54.931106 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:54.948941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:54.950144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:54.970382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:55.010486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1285/2538] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:55.020472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:55.022142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:55.090855 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01568e8f-a14a-4864-9977-6c7860e358e3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:55.095074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb1157c-ad9a-4afe-b6ec-7028ed414bed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:55.095728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1157c-ad9a-4afe-b6ec-7028ed414bed service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:55.096005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1157c-ad9a-4afe-b6ec-7028ed414bed service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:55.096303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb1157c-ad9a-4afe-b6ec-7028ed414bed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:55.109342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb1157c-ad9a-4afe-b6ec-7028ed414bed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:55.111682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb1157c-ad9a-4afe-b6ec-7028ed414bed service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:55.124118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1254/2539] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:13:56.189772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:56.191356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:56.206892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aea0c3f-0df3-40fc-b207-32b9b34a3ae6 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:56.218688 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:56.219695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:56.220809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:56.221098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] PATH_INFO: `/v3/users/3aee95220e114ab3ab48b74873d705b6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:56.223219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3aee95220e114ab3ab48b74873d705b6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:56.225920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:56.299398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3aee95220e114ab3ab48b74873d705b6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:56.300131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3aee95220e114ab3ab48b74873d705b6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:56.310654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3aee95220e114ab3ab48b74873d705b6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:56.312372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3aee95220e114ab3ab48b74873d705b6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:56.323954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3aee95220e114ab3ab48b74873d705b6'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:56.356899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1286/2540] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:13:56 2022] DELETE /identity/v3/users/3aee95220e114ab3ab48b74873d705b6 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:13:56.361592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:56.362472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:56.362935 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:56.383882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:13:56.385946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:56.473586 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1255/2541] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:13:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:13:56.485587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:56.487357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:56.596459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f67be0c-b65f-4ae4-bd21-e65bb2cc5c41 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:56.601026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b792a344-5f23-4929-a482-5648cc516cf1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:56.601553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b792a344-5f23-4929-a482-5648cc516cf1 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:56.601798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b792a344-5f23-4929-a482-5648cc516cf1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:56.602126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b792a344-5f23-4929-a482-5648cc516cf1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:56.616497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b792a344-5f23-4929-a482-5648cc516cf1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:56.618041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b792a344-5f23-4929-a482-5648cc516cf1 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:56.635963 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1287/2542] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:13:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:13:57.325183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:13:57.326622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:13:57.347062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db5a4e-8340-4830-ab5a-5cac62faba80 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:13:57.356233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:13:57.358604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:13:57.358894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:13:57.359173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] PATH_INFO: `/v3/projects/ee432e64b3c84f3b83227bfa25e4293b` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:13:57.360609 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee432e64b3c84f3b83227bfa25e4293b)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:13:57.362789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:13:57.419990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:13:57.426153 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee432e64b3c84f3b83227bfa25e4293b', 'request_id': 'req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:57.434885 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee432e64b3c84f3b83227bfa25e4293b', 'request_id': 'req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:57.442843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee432e64b3c84f3b83227bfa25e4293b', 'request_id': 'req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:57.458570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] The token cache is being invalidate because project ee432e64b3c84f3b83227bfa25e4293b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:13:57.459010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:57.459634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:13:57.468498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1256/2543] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:13:57 2022] DELETE /identity/v3/projects/ee432e64b3c84f3b83227bfa25e4293b => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:14:00.003058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66078f0-d0d0-47fe-9e45-c62899977f08 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:00.003559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66078f0-d0d0-47fe-9e45-c62899977f08 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:00.003652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66078f0-d0d0-47fe-9e45-c62899977f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:00.036625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c66078f0-d0d0-47fe-9e45-c62899977f08 None None] MFA Rules not processed for user `f0351b9ab5ac4fbaad38df073518412a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:00.038074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c66078f0-d0d0-47fe-9e45-c62899977f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:00.070361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:00.110560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1288/2544] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:00.115191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:00.116700 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:00.182423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf22a7b-9423-4a0d-a05c-f17b118dbb99 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:00.187214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:00.187931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:00.188163 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:00.188520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:00.189235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:00.191170 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:00.220258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1257/2545] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:14:00 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:14:00.224363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66078f0-d0d0-47fe-9e45-c62899977f08 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:00.236187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66078f0-d0d0-47fe-9e45-c62899977f08 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:00.300033 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:00.300748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:00.301030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:00.301307 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:00.302028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:00.303955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:00.324983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1289/2546] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:14:00 2022] POST /identity/v3/users => generated 299 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:14:00.329042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:00.340992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfaaf270-8371-486c-ac58-a3d5ca4e89d4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:00.346390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:00.347076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:00.347358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:00.347634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:00.348369 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:00.350061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:00.363611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1258/2547] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:14:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:14:00.367870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:00.378364 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d43ad-87e4-4312-9471-86f38fe2ee8e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:00.383670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:00.384596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:00.384880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:00.385157 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] PATH_INFO: `/v3/projects/50fc476f36f14020832aa6e34fe9146a/users/bf24995b6e0b48d1827357265ab34f7e/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:00.408592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] RBAC: Authorizing `identity:create_grant(project_id=50fc476f36f14020832aa6e34fe9146a, user_id=bf24995b6e0b48d1827357265ab34f7e, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:00.408592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:00.431120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1290/2548] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:14:00 2022] PUT /identity/v3/projects/50fc476f36f14020832aa6e34fe9146a/users/bf24995b6e0b48d1827357265ab34f7e/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:14:00.432768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:00.443254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd480796-5e11-49ec-acf6-331ce52e7d3f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:00.510600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cfb339-93fb-4725-a786-e5b8c52f10c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:00.511288 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cfb339-93fb-4725-a786-e5b8c52f10c8 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:00.511575 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cfb339-93fb-4725-a786-e5b8c52f10c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:00.511854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cfb339-93fb-4725-a786-e5b8c52f10c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:00.512645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cfb339-93fb-4725-a786-e5b8c52f10c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:00.514455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5cfb339-93fb-4725-a786-e5b8c52f10c8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:00.525383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1259/2549] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:14:00 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:14:00.529609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:00.542889 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9e79ac-0358-4a6c-a431-07f9a9b7e98a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:00.547978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:00.549010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:00.549372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:00.549850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] PATH_INFO: `/v3/projects/50fc476f36f14020832aa6e34fe9146a/users/bf24995b6e0b48d1827357265ab34f7e/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:00.552142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] RBAC: Authorizing `identity:create_grant(project_id=50fc476f36f14020832aa6e34fe9146a, user_id=bf24995b6e0b48d1827357265ab34f7e, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:00.555478 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:00.567319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1291/2550] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:14:00 2022] PUT /identity/v3/projects/50fc476f36f14020832aa6e34fe9146a/users/bf24995b6e0b48d1827357265ab34f7e/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:14:05.197224 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c89d268-5678-4d8b-ba05-74b4baee071c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:05.197721 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c89d268-5678-4d8b-ba05-74b4baee071c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:05.197843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c89d268-5678-4d8b-ba05-74b4baee071c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:05.226995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-2c89d268-5678-4d8b-ba05-74b4baee071c None None] MFA Rules not processed for user `bf24995b6e0b48d1827357265ab34f7e`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:05.228861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2c89d268-5678-4d8b-ba05-74b4baee071c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:05.262321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:05.323599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1260/2551] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:05 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:05.335692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:05.338690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:05.425099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddca1798-a886-446b-b572-202485fdf54b admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:05.429922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:05.430823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:05.431061 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:05.431340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:05.450823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:05.499326 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:05.500860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:05.511425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1292/2552] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:14:05 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:14:05.625555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01a6005-b246-497f-b194-46fa43767731 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:05.625980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01a6005-b246-497f-b194-46fa43767731 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:05.626335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01a6005-b246-497f-b194-46fa43767731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:05.649776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-e01a6005-b246-497f-b194-46fa43767731 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:05.651132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e01a6005-b246-497f-b194-46fa43767731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:05.716412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1261/2553] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:14:05 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:05.741084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:05.743557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:05.847499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed5276-ce87-4027-a0f0-9d4e7966ebdd service glance] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:05.854044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:05.854771 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:05.855058 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:05.855433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:05.874914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:05.880255 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:05.893564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1293/2554] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:14:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:14:15.708116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=39 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:15.855757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa34d537-efff-4ec5-8d3c-5d070ee65977 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:15.856094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa34d537-efff-4ec5-8d3c-5d070ee65977 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:15.856324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa34d537-efff-4ec5-8d3c-5d070ee65977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:15.873693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fa34d537-efff-4ec5-8d3c-5d070ee65977 None None] MFA Rules not processed for user `f0351b9ab5ac4fbaad38df073518412a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:15.874920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fa34d537-efff-4ec5-8d3c-5d070ee65977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:15.882002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1262/2555] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:15.888791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=51 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:16.294971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:16.298063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:16.366888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889240d-039b-49fd-848c-09114afffde9 service swift] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:16.371296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:16.371945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:16.372229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:16.372550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:16.374350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:16.445357 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:16.447468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:16.457358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1294/2556] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:14:16 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:14:18.275060 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:18.275573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:18.275682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:18.294684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:18.296184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:18.312319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1263/2557] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:14:18 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:18.327885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:18.342072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925e92fe-c868-4e02-9ff4-04b435df9796 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:18.348819 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5452a08-0523-4ae8-9d4e-0ff444c94605 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:18.349512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5452a08-0523-4ae8-9d4e-0ff444c94605 service swift] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:18.349799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5452a08-0523-4ae8-9d4e-0ff444c94605 service swift] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:18.350074 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5452a08-0523-4ae8-9d4e-0ff444c94605 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:18.359888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5452a08-0523-4ae8-9d4e-0ff444c94605 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:18.361526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5452a08-0523-4ae8-9d4e-0ff444c94605 service swift] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:18.370806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1295/2558] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:14:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:14:28.289199 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=6 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:28.367534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=56 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:49.487437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:49.488950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:49.561459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f3cee3-901a-46b4-9215-acefdefa8a80 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:49.566096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:49.566970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:49.567251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:49.567525 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] PATH_INFO: `/v3/users/572788c3d7a24fe4afe6b223801f3192` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:49.568755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] RBAC: Authorizing `identity:delete_user(user_id=572788c3d7a24fe4afe6b223801f3192)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:49.570677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:49.615395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '572788c3d7a24fe4afe6b223801f3192'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:49.622710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '572788c3d7a24fe4afe6b223801f3192'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:49.623371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '572788c3d7a24fe4afe6b223801f3192'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:49.623981 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '572788c3d7a24fe4afe6b223801f3192'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:49.635764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '572788c3d7a24fe4afe6b223801f3192'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:49.648119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1264/2559] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:14:49 2022] DELETE /identity/v3/users/572788c3d7a24fe4afe6b223801f3192 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:14:49.652732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:49.653263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:49.653263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:49.670614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:49.671797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:49.728903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1296/2560] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:49.738614 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:49.740044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:49.796753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5796d4d-706e-4584-8609-43c34db3974d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:49.800286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9511cff7-6f3a-4944-bcbc-05f5b4a9fe8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:49.800985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9511cff7-6f3a-4944-bcbc-05f5b4a9fe8a service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:49.801271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9511cff7-6f3a-4944-bcbc-05f5b4a9fe8a service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:49.801551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9511cff7-6f3a-4944-bcbc-05f5b4a9fe8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:49.810530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9511cff7-6f3a-4944-bcbc-05f5b4a9fe8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:49.812111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9511cff7-6f3a-4944-bcbc-05f5b4a9fe8a service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:49.822103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1265/2561] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:14:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:14:50.114811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.116293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:50.129398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f3ff4d-c2aa-4746-913a-e72f56c335d4 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.139665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.141918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.142559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.143497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] PATH_INFO: `/v3/projects/e008272f22d74b0e97fdc329a3ec7763` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.145783 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e008272f22d74b0e97fdc329a3ec7763)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.147950 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.222646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e008272f22d74b0e97fdc329a3ec7763', 'request_id': 'req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:50.231472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e008272f22d74b0e97fdc329a3ec7763', 'request_id': 'req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:50.240732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e008272f22d74b0e97fdc329a3ec7763', 'request_id': 'req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:50.254633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] The token cache is being invalidate because project e008272f22d74b0e97fdc329a3ec7763 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:14:50.255041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:50.255854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1069f1c0-0567-49ed-82c4-834ee9f7ae1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:50.257716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1297/2562] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:14:50 2022] DELETE /identity/v3/projects/e008272f22d74b0e97fdc329a3ec7763 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:14:50.264312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5e970-9640-4f40-8449-a1b461c0d998 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.264745 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5e970-9640-4f40-8449-a1b461c0d998 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.265008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5e970-9640-4f40-8449-a1b461c0d998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.285570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b6a5e970-9640-4f40-8449-a1b461c0d998 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:50.286828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b6a5e970-9640-4f40-8449-a1b461c0d998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:50.306832 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:50.356839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1266/2563] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:50.364273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af07ef7-a8f4-41c3-b009-a8c320c9b12a None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.364594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af07ef7-a8f4-41c3-b009-a8c320c9b12a None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.364879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af07ef7-a8f4-41c3-b009-a8c320c9b12a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.383138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-2af07ef7-a8f4-41c3-b009-a8c320c9b12a None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:50.384316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2af07ef7-a8f4-41c3-b009-a8c320c9b12a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:50.392062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1298/2564] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:50.395960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5e970-9640-4f40-8449-a1b461c0d998 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.405487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5e970-9640-4f40-8449-a1b461c0d998 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.409979 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.410677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.410913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.411191 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.411918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:14:50.412228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.413864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.425819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1267/2565] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:14:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:14:50.429911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af07ef7-a8f4-41c3-b009-a8c320c9b12a None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.439603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af07ef7-a8f4-41c3-b009-a8c320c9b12a None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.444429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.445072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.445351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.445623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.446302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.448434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.473304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1299/2566] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:14:50 2022] POST /identity/v3/projects => generated 371 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:14:50.477406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.487980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e871fc9b-def8-44c3-8176-1edea93a0df4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.551423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.552146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.552481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.552768 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.553530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.557055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.578241 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1268/2567] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:14:50 2022] POST /identity/v3/users => generated 286 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:14:50.581059 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.591606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3ca801-208c-401d-b9b3-3a3afe28f028 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.596671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.597359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.597662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.597942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.600271 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.612430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1300/2568] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:14:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:14:50.616355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.627203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb19d76a-3923-4ca6-9c46-e757bb804740 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.632266 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e3827-18c7-4fc2-a442-0eb5261daaf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.632991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e3827-18c7-4fc2-a442-0eb5261daaf6 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.633275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e3827-18c7-4fc2-a442-0eb5261daaf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.633552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e3827-18c7-4fc2-a442-0eb5261daaf6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.634276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e3827-18c7-4fc2-a442-0eb5261daaf6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.635912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e3827-18c7-4fc2-a442-0eb5261daaf6 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.646277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1269/2569] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:14:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:14:50.650400 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.660997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6eaeac-b302-47be-a7d9-c98e4b197882 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.665559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.666439 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.666719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.666995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] PATH_INFO: `/v3/projects/5b1901d2de20431a8c8ddf4311f1868b/users/8ef1a08c3801450fa9cc8d134b296047/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.682374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5b1901d2de20431a8c8ddf4311f1868b, user_id=8ef1a08c3801450fa9cc8d134b296047, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.685876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.701091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1301/2570] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:14:50 2022] PUT /identity/v3/projects/5b1901d2de20431a8c8ddf4311f1868b/users/8ef1a08c3801450fa9cc8d134b296047/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:14:50.713532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f776d08-f833-481a-8d7e-000511405e19 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.713829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f776d08-f833-481a-8d7e-000511405e19 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.714115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f776d08-f833-481a-8d7e-000511405e19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.743274 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-6f776d08-f833-481a-8d7e-000511405e19 None None] MFA Rules not processed for user `8ef1a08c3801450fa9cc8d134b296047`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:50.744422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6f776d08-f833-481a-8d7e-000511405e19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:50.813696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1270/2571] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:50 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:50.827667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.829647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:50.852334 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:50.891811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7719cb6-157e-4a49-8fd2-c3cabc492921 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:50.895424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e11111-361e-4625-823a-52f4c2e4b4a3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:50.896065 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e11111-361e-4625-823a-52f4c2e4b4a3 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:50.896348 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e11111-361e-4625-823a-52f4c2e4b4a3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:50.896661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e11111-361e-4625-823a-52f4c2e4b4a3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:50.905651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e11111-361e-4625-823a-52f4c2e4b4a3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:50.907084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e11111-361e-4625-823a-52f4c2e4b4a3 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:50.915810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1302/2572] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:14:50 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:14:50.941916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f776d08-f833-481a-8d7e-000511405e19 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:50.961122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:14:51.010179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f776d08-f833-481a-8d7e-000511405e19 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:51.018076 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:51.018953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:51.019239 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:51.019517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] PATH_INFO: `/v3/users/8ef1a08c3801450fa9cc8d134b296047` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:51.020773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ef1a08c3801450fa9cc8d134b296047)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:51.022719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:51.071441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ef1a08c3801450fa9cc8d134b296047'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.082236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ef1a08c3801450fa9cc8d134b296047'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.082926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ef1a08c3801450fa9cc8d134b296047'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.083538 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ef1a08c3801450fa9cc8d134b296047'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.102029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ef1a08c3801450fa9cc8d134b296047'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.115084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1271/2573] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:14:50 2022] DELETE /identity/v3/users/8ef1a08c3801450fa9cc8d134b296047 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:14:51.119565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:51.119855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:51.120132 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:51.138442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:14:51.139655 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:51.196080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1303/2574] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:14:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:14:51.206063 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:51.207487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:51.265416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e3057-b42c-4d26-bf8f-508fec3d0432 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:51.269217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d6ccf5-f698-4e69-9131-b6000d5b1862 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:51.269760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d6ccf5-f698-4e69-9131-b6000d5b1862 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:51.270049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d6ccf5-f698-4e69-9131-b6000d5b1862 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:51.270333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d6ccf5-f698-4e69-9131-b6000d5b1862 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:51.279668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d6ccf5-f698-4e69-9131-b6000d5b1862 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:51.281138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d6ccf5-f698-4e69-9131-b6000d5b1862 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:51.291814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1272/2575] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:14:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:14:51.559503 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:14:51.561034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:14:51.574652 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4265ff-bdb4-4290-98d9-859e6101d329 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:14:51.579371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:14:51.580248 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:14:51.580564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:14:51.580871 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] PATH_INFO: `/v3/projects/5b1901d2de20431a8c8ddf4311f1868b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:14:51.582070 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5b1901d2de20431a8c8ddf4311f1868b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:14:51.584128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:14:51.631886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5b1901d2de20431a8c8ddf4311f1868b', 'request_id': 'req-d1bd1250-2605-4fa5-b293-c24d8339700a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.637485 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5b1901d2de20431a8c8ddf4311f1868b', 'request_id': 'req-d1bd1250-2605-4fa5-b293-c24d8339700a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.644671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5b1901d2de20431a8c8ddf4311f1868b', 'request_id': 'req-d1bd1250-2605-4fa5-b293-c24d8339700a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.656680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] The token cache is being invalidate because project 5b1901d2de20431a8c8ddf4311f1868b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:14:51.656969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.657599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:14:51.659147 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1304/2576] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:14:51 2022] DELETE /identity/v3/projects/5b1901d2de20431a8c8ddf4311f1868b => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:15:01.286824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=71,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:15:01.653596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=71,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:15:03.165265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:03.165800 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:03.165897 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:03.183257 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] MFA Rules not processed for user `036cb887c7894e75ba3eb4c1a8b21c69`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:15:03.184505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:03.238165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1273/2577] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:15:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:15:06.026604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:06.028024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:06.090037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd1250-2605-4fa5-b293-c24d8339700a admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:06.094586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:06.095374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:06.095768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:06.096040 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:06.097984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:06.140251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:06.142476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:06.194109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:06.195658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:06.206278 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ad6460-3fa2-4fc9-9033-9e950c0f420b None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:06.209966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:06.210618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:06.210905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:06.211213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:06.213044 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:06.220719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1305/2578] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:15:06.227252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:06.228724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:06.241265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1274/2579] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:15:16.175516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:16.176967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:16.188968 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd52048-6d14-4c1f-a24c-1de8137e1620 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:16.192695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:16.193341 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:16.193628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:16.193911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:16.195712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:16.236851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=51 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:15:16.270017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:16.271495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:16.282388 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1306/2580] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:16 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:15:19.143010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:19.144472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:19.158210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd4553-070d-42cb-af03-7c3054d4f0c8 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:19.163659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be6bbf6-428a-40b6-bc3f-b17800f22913 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:19.164299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be6bbf6-428a-40b6-bc3f-b17800f22913 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:19.164637 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be6bbf6-428a-40b6-bc3f-b17800f22913 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:19.164918 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be6bbf6-428a-40b6-bc3f-b17800f22913 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:19.166846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5be6bbf6-428a-40b6-bc3f-b17800f22913 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:19.177679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be6bbf6-428a-40b6-bc3f-b17800f22913 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:19.179321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be6bbf6-428a-40b6-bc3f-b17800f22913 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:19.189357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1275/2581] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:19 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:15:25.933097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:25.941813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db6413c-0241-467b-ad6d-db7c8602562d service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:25.945429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:25.946060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:25.946340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:25.946615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:25.948327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:25.957777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:25.959194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:25.967499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1307/2582] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:25 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:15:29.184915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=6 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:15:34.935753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:34.936324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:34.936324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:34.953573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] MFA Rules not processed for user `df495d1fbab24736a2ba12e8a0e7d803`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:15:34.954772 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:35.009682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1276/2583] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:15:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:15:35.244108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:35.248256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=100 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:15:35.254197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0eab5f-f4b0-4d00-b60d-e0ec706c5f6d service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:35.257846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:35.258511 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:35.258799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:35.259085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:35.260882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:35.270848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:35.272265 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:35.280785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1308/2584] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:35 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:15:35.580464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:35.581865 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:35.653263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfae60eb-937e-4b01-afdf-4c51ddeab4b5 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:35.658252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:35.658901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:35.659190 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:35.659475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:35.661276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:35.672518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:35.674108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:35.683947 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1277/2585] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:35 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:15:38.086302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:15:38.087721 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:38.104109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f384bf-8488-430b-bfe2-dab02aba0343 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:15:38.109875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:15:38.110667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:15:38.110843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:15:38.111085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:15:38.112960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:15:38.126144 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:15:38.127720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:15:38.138687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1309/2586] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:15:38 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:15:45.679729 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=57 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:15:48.133845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=11 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:16:18.084137 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:16:18.096764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dc8667-d123-4ca7-a285-aa8e8c6df2e4 service placement] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:16:18.103234 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:16:18.104071 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:16:18.104221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:16:18.104451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:16:18.106231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:16:18.166676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:16:18.168308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:16:18.177727 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1278/2587] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:16:18 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:16:28.173879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=28 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:16:46.125879 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:16:46.127372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:16:46.195812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6252cf-9475-475d-847a-613e1d2f63dc service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:16:46.200611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:16:46.202210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:16:46.202498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:16:46.202782 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] PATH_INFO: `/v3/users/df495d1fbab24736a2ba12e8a0e7d803` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:16:46.203984 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] RBAC: Authorizing `identity:delete_user(user_id=df495d1fbab24736a2ba12e8a0e7d803)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:16:46.206007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:16:46.257693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'df495d1fbab24736a2ba12e8a0e7d803'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.258358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'df495d1fbab24736a2ba12e8a0e7d803'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.265988 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'df495d1fbab24736a2ba12e8a0e7d803'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.266634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'df495d1fbab24736a2ba12e8a0e7d803'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.273525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1106a1d0-e106-4ac1-910a-8d3b246b428a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'df495d1fbab24736a2ba12e8a0e7d803'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.290602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1310/2588] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:16:46 2022] DELETE /identity/v3/users/df495d1fbab24736a2ba12e8a0e7d803 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:16:46.465821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:16:46.479036 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3080fe01-e7cd-422c-a3dc-72e0b9964442 service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:16:46.545751 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:16:46.547127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:16:46.547408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:16:46.547686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] PATH_INFO: `/v3/projects/accabbba722345d190b52446d1e7c2b7` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:16:46.548982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] RBAC: Authorizing `identity:delete_project(project_id=accabbba722345d190b52446d1e7c2b7)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:16:46.551041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:16:46.598673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'accabbba722345d190b52446d1e7c2b7', 'request_id': 'req-2ea181f6-f096-44a0-8195-c60cf827e745'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.604616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'accabbba722345d190b52446d1e7c2b7', 'request_id': 'req-2ea181f6-f096-44a0-8195-c60cf827e745'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.611532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'accabbba722345d190b52446d1e7c2b7', 'request_id': 'req-2ea181f6-f096-44a0-8195-c60cf827e745'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.625267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] The token cache is being invalidate because project accabbba722345d190b52446d1e7c2b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:16:46.625605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.626246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2ea181f6-f096-44a0-8195-c60cf827e745 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:16:46.628088 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1279/2589] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:16:46 2022] DELETE /identity/v3/projects/accabbba722345d190b52446d1e7c2b7 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:16:48.587411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:16:48.587970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:16:48.587970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:16:48.607775 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] MFA Rules not processed for user `248946c73cb245f9b8e6fe73e46133cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:16:48.609639 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:16:48.662895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1311/2590] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:16:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:16:56.621373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=51,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:16:58.656821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=78,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:17:34.575272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:34.575799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:34.575908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:34.593512 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] MFA Rules not processed for user `036cb887c7894e75ba3eb4c1a8b21c69`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:17:34.594726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:34.648262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1280/2591] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:17:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:17:34.941935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:17:34.943906 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:35.000316 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e0b0ca-6097-4848-b82c-4b138279aa0d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:17:35.003872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:17:35.004550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:35.004835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:35.005115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:35.007377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:35.016685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:17:35.018151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:17:35.026977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1312/2592] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:17:34 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:17:41.326636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:17:41.328549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:41.417314 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9242ea06-41d0-43e8-b0ea-04a06ebefea7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:17:41.423678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008c9fa1-8fda-41c3-a911-86715b2ea35a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:17:41.424608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c9fa1-8fda-41c3-a911-86715b2ea35a service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:41.425086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c9fa1-8fda-41c3-a911-86715b2ea35a service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:41.425627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008c9fa1-8fda-41c3-a911-86715b2ea35a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:41.428074 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-008c9fa1-8fda-41c3-a911-86715b2ea35a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:41.499179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c9fa1-8fda-41c3-a911-86715b2ea35a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:17:41.501625 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008c9fa1-8fda-41c3-a911-86715b2ea35a service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:17:41.516091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1281/2593] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:17:41 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:17:43.465042 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:17:43.466546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:43.485184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9a5ddc-ca62-4fed-b841-b45bc18cdf65 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:17:43.490597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5948f0-8089-4065-9781-f2b1fe8b55e2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:17:43.491276 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5948f0-8089-4065-9781-f2b1fe8b55e2 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:43.491555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5948f0-8089-4065-9781-f2b1fe8b55e2 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:43.491830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5948f0-8089-4065-9781-f2b1fe8b55e2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:43.493634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1d5948f0-8089-4065-9781-f2b1fe8b55e2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:43.506722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5948f0-8089-4065-9781-f2b1fe8b55e2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:17:43.508301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5948f0-8089-4065-9781-f2b1fe8b55e2 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:17:43.519161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1313/2594] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:17:43 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:17:46.719548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9836b96-1436-4c29-8cee-dfea16323dbf None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:46.720101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9836b96-1436-4c29-8cee-dfea16323dbf None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:46.721286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9836b96-1436-4c29-8cee-dfea16323dbf None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:46.722956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1282/2595] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:17:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:17:46.728821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9e3faa-37e7-41d2-ad95-24fd2ba2a6b5 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:46.729616 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9e3faa-37e7-41d2-ad95-24fd2ba2a6b5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:46.729866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9e3faa-37e7-41d2-ad95-24fd2ba2a6b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:46.749947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4d9e3faa-37e7-41d2-ad95-24fd2ba2a6b5 None None] MFA Rules not processed for user `36ce9bd8ce6448b8b09fe9eb7b8f7c54`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:17:46.751549 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4d9e3faa-37e7-41d2-ad95-24fd2ba2a6b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:46.758661 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1314/2596] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:17:46 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:17:46.771208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9836b96-1436-4c29-8cee-dfea16323dbf None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:17:46.773387 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a9836b96-1436-4c29-8cee-dfea16323dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:46.786868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9836b96-1436-4c29-8cee-dfea16323dbf None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:17:46.790964 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11e1073-3a01-41b1-a657-c333ecf7c0a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:17:46.791167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e1073-3a01-41b1-a657-c333ecf7c0a9 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:46.791584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e1073-3a01-41b1-a657-c333ecf7c0a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:46.791931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11e1073-3a01-41b1-a657-c333ecf7c0a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:46.801396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11e1073-3a01-41b1-a657-c333ecf7c0a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:17:46.803900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11e1073-3a01-41b1-a657-c333ecf7c0a9 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:17:46.813544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1283/2597] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:17:46 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:17:46.899233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c80483-30ae-43b3-92ad-1178bb439aef None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:46.899620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c80483-30ae-43b3-92ad-1178bb439aef None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:46.899969 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c80483-30ae-43b3-92ad-1178bb439aef None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:46.901474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1315/2598] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:17:46.907323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d8d4d6-c6e8-4c62-a05d-57bed983ad89 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:46.907775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d8d4d6-c6e8-4c62-a05d-57bed983ad89 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:46.908070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d8d4d6-c6e8-4c62-a05d-57bed983ad89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:46.953902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-09d8d4d6-c6e8-4c62-a05d-57bed983ad89 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:17:46.955151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-09d8d4d6-c6e8-4c62-a05d-57bed983ad89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:17:46.965958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1284/2599] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:17:46 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:17:46.977272 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c80483-30ae-43b3-92ad-1178bb439aef None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:17:46.991270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c80483-30ae-43b3-92ad-1178bb439aef None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:17:46.994913 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:17:46.995610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:17:46.995893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:17:46.996172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:17:47.007361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:17:47.008855 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:17:47.020472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1316/2600] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:17:46 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:17:56.950806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:17:57.015312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=41 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:17.159763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d8d4d6-c6e8-4c62-a05d-57bed983ad89 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:17.169180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d8d4d6-c6e8-4c62-a05d-57bed983ad89 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:17.172744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:17.173388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:17.173678 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:17.173963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:17.175734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:17.241254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:17.242611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:17.253143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1285/2601] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:18:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:18:20.475874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:20.477358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:20.543275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:20.544871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:20.557122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806eeef3-c033-4e91-ba07-3fcc07a2a046 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:20.559660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d99e8-0967-4981-b4ab-4397dfe98773 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:20.561622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:20.562493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:20.562779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:20.563054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] PATH_INFO: `/v3/users/248946c73cb245f9b8e6fe73e46133cf` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:20.564228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=248946c73cb245f9b8e6fe73e46133cf)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:20.565584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:20.566231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:20.566519 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:20.566802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:20.568592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:20.575022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:20.581159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:20.582776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:20.593650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1317/2602] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:18:20 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:18:20.635899 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '248946c73cb245f9b8e6fe73e46133cf'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.643086 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '248946c73cb245f9b8e6fe73e46133cf'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.643750 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '248946c73cb245f9b8e6fe73e46133cf'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.644348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '248946c73cb245f9b8e6fe73e46133cf'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.656822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-38a77d52-3edd-458c-a83b-9c5f4bb9ccbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '248946c73cb245f9b8e6fe73e46133cf'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.669185 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1286/2603] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:18:20 2022] DELETE /identity/v3/users/248946c73cb245f9b8e6fe73e46133cf => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:18:20.840986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:20.852169 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a0123-d556-4b09-97fb-3589d864c005 service placement] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:20.908294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:20.909174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:20.909454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:20.909726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] PATH_INFO: `/v3/projects/3f2199fa0bc54901aa24195a250a9e25` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:20.910885 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f2199fa0bc54901aa24195a250a9e25)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:20.912826 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:20.959300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f2199fa0bc54901aa24195a250a9e25', 'request_id': 'req-e2d2f0ef-ec95-47ee-b987-1336fe666510'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.964966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f2199fa0bc54901aa24195a250a9e25', 'request_id': 'req-e2d2f0ef-ec95-47ee-b987-1336fe666510'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.971845 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f2199fa0bc54901aa24195a250a9e25', 'request_id': 'req-e2d2f0ef-ec95-47ee-b987-1336fe666510'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.984225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] The token cache is being invalidate because project 3f2199fa0bc54901aa24195a250a9e25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:18:20.984225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.984822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d2f0ef-ec95-47ee-b987-1336fe666510 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:20.986427 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1318/2604] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:18:20 2022] DELETE /identity/v3/projects/3f2199fa0bc54901aa24195a250a9e25 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:18:20.992309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb95f2f-ee77-4acc-b6a5-4d2efd1b26ec None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:20.992629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb95f2f-ee77-4acc-b6a5-4d2efd1b26ec None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:20.992911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb95f2f-ee77-4acc-b6a5-4d2efd1b26ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.013733 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-afb95f2f-ee77-4acc-b6a5-4d2efd1b26ec None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:18:21.015049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-afb95f2f-ee77-4acc-b6a5-4d2efd1b26ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:21.078059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1287/2605] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:18:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:18:21.086233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f929b15d-0386-4678-87fa-b8c2705a623d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.086527 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f929b15d-0386-4678-87fa-b8c2705a623d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.086929 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f929b15d-0386-4678-87fa-b8c2705a623d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.104931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f929b15d-0386-4678-87fa-b8c2705a623d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:18:21.106150 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f929b15d-0386-4678-87fa-b8c2705a623d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:21.116815 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1319/2606] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:18:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:18:21.119524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb95f2f-ee77-4acc-b6a5-4d2efd1b26ec None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:21.129652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb95f2f-ee77-4acc-b6a5-4d2efd1b26ec None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:21.134856 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:21.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.135809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.136090 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.136896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:18:21.137206 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:21.138816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:21.140272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:21.152720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1288/2607] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:18:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:18:21.156970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f929b15d-0386-4678-87fa-b8c2705a623d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:21.166131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f929b15d-0386-4678-87fa-b8c2705a623d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:21.170724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:21.171365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.171646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.171922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.172631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:21.174516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:21.184497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:21.210841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1320/2608] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:18:21 2022] POST /identity/v3/projects => generated 389 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:18:21.214906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:21.224841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741bfca8-ae7e-41d9-b50f-2118bd07901f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:21.286732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:21.287404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.287692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.287975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.288689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:21.291243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:21.311539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1289/2609] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:18:21 2022] POST /identity/v3/users => generated 295 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:18:21.315599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:21.326949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f429b2-54ab-47a3-8980-02b62bd8fbb1 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:21.331596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:21.332251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.332634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.332914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.333638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:21.335235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:21.345422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1321/2610] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:18:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:18:21.349327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:21.359760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f543db2-0985-4042-80b0-3ba9400ec469 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:21.364627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5ed86f-a0e1-4208-8eb2-9c3b6c294e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:21.365276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ed86f-a0e1-4208-8eb2-9c3b6c294e4b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.365557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ed86f-a0e1-4208-8eb2-9c3b6c294e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.365831 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ed86f-a0e1-4208-8eb2-9c3b6c294e4b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.366548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5ed86f-a0e1-4208-8eb2-9c3b6c294e4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:21.368123 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5ed86f-a0e1-4208-8eb2-9c3b6c294e4b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:21.378469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1290/2611] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:18:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:18:21.382296 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:21.392167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacbd6ab-f858-4ffe-b835-7aa7110fba22 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:21.396715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:21.397581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.397860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.398131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] PATH_INFO: `/v3/projects/f0c19d009f1b4ef09a40267bcf654107/users/d4a608563189457dbcadf8d48acdd84a/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.412447 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0c19d009f1b4ef09a40267bcf654107, user_id=d4a608563189457dbcadf8d48acdd84a, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:21.415083 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:21.428046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1322/2612] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:18:21 2022] PUT /identity/v3/projects/f0c19d009f1b4ef09a40267bcf654107/users/d4a608563189457dbcadf8d48acdd84a/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:18:21.433714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471bff4-abdb-40c8-993e-b022212adcd7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.434039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471bff4-abdb-40c8-993e-b022212adcd7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.434355 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471bff4-abdb-40c8-993e-b022212adcd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.451857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f471bff4-abdb-40c8-993e-b022212adcd7 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:18:21.453103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f471bff4-abdb-40c8-993e-b022212adcd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:21.511900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1291/2613] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:18:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:18:21.521343 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:21.523010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:21.577915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d7894a-2297-4fef-a8cf-1439db97ef87 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:21.581445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:21.582080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:21.582358 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:21.582631 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:21.591887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:21.593324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:21.602500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1323/2614] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:18:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:18:25.427956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5834c2-8947-46f9-892b-cab70d7dddc4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:25.428473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5834c2-8947-46f9-892b-cab70d7dddc4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:25.428591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5834c2-8947-46f9-892b-cab70d7dddc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:25.458759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-3c5834c2-8947-46f9-892b-cab70d7dddc4 None None] MFA Rules not processed for user `d4a608563189457dbcadf8d48acdd84a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:18:25.460618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-3c5834c2-8947-46f9-892b-cab70d7dddc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:25.529707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1292/2615] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:18:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:18:25.545210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:25.546844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:25.625351 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac45767-8830-4023-bef9-9a89ad41353d service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:25.630996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:25.631660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:25.631942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:25.632217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:25.641151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:25.642851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:25.652492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1324/2616] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:18:25 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:18:26.999987 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:27.000483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:27.000578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:27.007803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:27.025577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] MFA Rules not processed for user `d4a608563189457dbcadf8d48acdd84a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:18:27.027286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:27.034401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1293/2617] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:18:26 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:18:28.155589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:28.157045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:28.198624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:28.228508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd093eb5-09b5-45f0-a280-5cae9dbfb25f service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:28.235936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b079467-80d8-4f2b-a9ac-0cf3869ccd2e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:28.236640 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b079467-80d8-4f2b-a9ac-0cf3869ccd2e service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:28.236930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b079467-80d8-4f2b-a9ac-0cf3869ccd2e service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:28.237204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b079467-80d8-4f2b-a9ac-0cf3869ccd2e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:28.239037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4b079467-80d8-4f2b-a9ac-0cf3869ccd2e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:28.252291 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b079467-80d8-4f2b-a9ac-0cf3869ccd2e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:28.254055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b079467-80d8-4f2b-a9ac-0cf3869ccd2e service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:28.264738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1325/2618] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:18:28 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:18:30.495905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:30.497634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:30.525249 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1b235b-1bdc-4ea6-b98e-da8f56249fb0 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:30.535362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:30.536970 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:30.537709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:30.538942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:30.543352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:30.571754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:30.577992 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:30.598121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1294/2619] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:18:30 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:18:38.259949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=16 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:39.111073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9db73d-9255-45c0-ab3d-99fd8b4be349 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:39.111362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9db73d-9255-45c0-ab3d-99fd8b4be349 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:39.111644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9db73d-9255-45c0-ab3d-99fd8b4be349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:39.129181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ce9db73d-9255-45c0-ab3d-99fd8b4be349 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:18:39.130373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ce9db73d-9255-45c0-ab3d-99fd8b4be349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:39.184434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1326/2620] 10.222.0.20 () {58 vars in 967 bytes} [Tue Jul 5 10:18:39 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:18:39.223148 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:39.224583 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:39.291969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10cce06-d562-4cf8-9a5d-214b470062fc service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:39.296904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:39.297542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:39.297828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:39.298108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:39.306066 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:39.307639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:39.315627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1295/2621] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:18:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:18:42.707437 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9db73d-9255-45c0-ab3d-99fd8b4be349 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:42.723273 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9db73d-9255-45c0-ab3d-99fd8b4be349 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:42.728262 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:42.728937 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:42.729218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:42.729492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:42.731860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:42.745005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:42.746654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:42.758776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1327/2622] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:18:42 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:18:46.103221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:46.104665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:46.119737 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298d604-b6d2-497c-b98d-9c3de2e4f705 service swift] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:46.125013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:46.125898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:46.126188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:46.126499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] PATH_INFO: `/v3/users/f0351b9ab5ac4fbaad38df073518412a` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:46.127702 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0351b9ab5ac4fbaad38df073518412a)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:46.129735 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:46.177061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0351b9ab5ac4fbaad38df073518412a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.184517 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0351b9ab5ac4fbaad38df073518412a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.185167 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0351b9ab5ac4fbaad38df073518412a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.185762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0351b9ab5ac4fbaad38df073518412a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.197347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0351b9ab5ac4fbaad38df073518412a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.210172 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1296/2623] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:18:46 2022] DELETE /identity/v3/users/f0351b9ab5ac4fbaad38df073518412a => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:18:46.379671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:46.390948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaad50e3-890b-4c38-bd28-b0829927d374 service placement] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:46.447697 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:46.448591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:46.448878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:46.449188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] PATH_INFO: `/v3/projects/d71ebc02c6094919b3054d7da84aadf0` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:46.450362 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d71ebc02c6094919b3054d7da84aadf0)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:46.452302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:46.499077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd71ebc02c6094919b3054d7da84aadf0', 'request_id': 'req-219b410f-b4ec-4b0a-8026-21c9e63c3d52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.505615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd71ebc02c6094919b3054d7da84aadf0', 'request_id': 'req-219b410f-b4ec-4b0a-8026-21c9e63c3d52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.512710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd71ebc02c6094919b3054d7da84aadf0', 'request_id': 'req-219b410f-b4ec-4b0a-8026-21c9e63c3d52'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.523765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] The token cache is being invalidate because project d71ebc02c6094919b3054d7da84aadf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:18:46.524048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.524674 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.526200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1328/2624] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:18:46 2022] DELETE /identity/v3/projects/d71ebc02c6094919b3054d7da84aadf0 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:18:46.530703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:46.532079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:46.545666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f1c7cf-aaff-4c85-8f02-765b0c7855ee admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:46.586009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:46.602908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:46.603795 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:46.604081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:46.604359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] PATH_INFO: `/v3/users/bf24995b6e0b48d1827357265ab34f7e` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:46.606230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf24995b6e0b48d1827357265ab34f7e)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:46.608188 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:46.656069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf24995b6e0b48d1827357265ab34f7e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.662774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf24995b6e0b48d1827357265ab34f7e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.663451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf24995b6e0b48d1827357265ab34f7e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.664050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf24995b6e0b48d1827357265ab34f7e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.676386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cb002e3c-9a75-4297-ad26-8d6cee24058f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf24995b6e0b48d1827357265ab34f7e'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.688993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1297/2625] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:18:46 2022] DELETE /identity/v3/users/bf24995b6e0b48d1827357265ab34f7e => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:18:46.861481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:18:46.862955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:18:46.880654 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219b410f-b4ec-4b0a-8026-21c9e63c3d52 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:18:46.900112 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:46.939609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:18:46.940503 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:18:46.940795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:18:46.941071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] PATH_INFO: `/v3/projects/50fc476f36f14020832aa6e34fe9146a` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:18:46.942251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=50fc476f36f14020832aa6e34fe9146a)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:18:46.944158 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:18:46.990379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50fc476f36f14020832aa6e34fe9146a', 'request_id': 'req-e0fd109a-be94-4ade-b00a-5570b3a48ec9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:46.995955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50fc476f36f14020832aa6e34fe9146a', 'request_id': 'req-e0fd109a-be94-4ade-b00a-5570b3a48ec9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:47.003719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50fc476f36f14020832aa6e34fe9146a', 'request_id': 'req-e0fd109a-be94-4ade-b00a-5570b3a48ec9'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:47.016766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] The token cache is being invalidate because project 50fc476f36f14020832aa6e34fe9146a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:18:47.017062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:47.017701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:18:47.019699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1329/2626] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:18:46 2022] DELETE /identity/v3/projects/50fc476f36f14020832aa6e34fe9146a => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:18:56.685983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=33,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:18:57.014041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=41,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:19:18.836764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:19:18.837293 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:19:18.837393 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:19:18.854886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] MFA Rules not processed for user `036cb887c7894e75ba3eb4c1a8b21c69`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:19:18.856094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:19:18.909453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1298/2627] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:19:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:19:28.902553 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=25 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:19:39.618233 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:19:39.619785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:19:39.682252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fd109a-be94-4ade-b00a-5570b3a48ec9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:19:39.685868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:19:39.686521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:19:39.686807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:19:39.687090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:19:39.689414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:19:39.753111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:19:39.754573 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:19:39.764136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1330/2628] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:19:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:19:45.926362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:19:45.927799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:19:45.943158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08197d55-43b5-4859-9f62-8ff78b623313 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:19:45.947662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:19:45.948570 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:19:45.948852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:19:45.949124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] PATH_INFO: `/v3/users/7e322227b42b4aaa99674afd906a706a` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:19:45.950299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e322227b42b4aaa99674afd906a706a)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:19:45.952213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:19:45.998572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e322227b42b4aaa99674afd906a706a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.006438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e322227b42b4aaa99674afd906a706a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.007105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e322227b42b4aaa99674afd906a706a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.007715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e322227b42b4aaa99674afd906a706a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.019386 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e322227b42b4aaa99674afd906a706a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.032068 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1299/2629] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:19:45 2022] DELETE /identity/v3/users/7e322227b42b4aaa99674afd906a706a => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:19:46.202253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:19:46.215029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b1c53a-5d07-42a2-9f24-e9d192f1fbe6 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:19:46.275461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:19:46.276335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:19:46.276656 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:19:46.276932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] PATH_INFO: `/v3/projects/9edab973767545988e178ddff76557ac` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:19:46.278115 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] RBAC: Authorizing `identity:delete_project(project_id=9edab973767545988e178ddff76557ac)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:19:46.280034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:19:46.327508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9edab973767545988e178ddff76557ac', 'request_id': 'req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.330375 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:19:46.335569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9edab973767545988e178ddff76557ac', 'request_id': 'req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.342977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9edab973767545988e178ddff76557ac', 'request_id': 'req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.353993 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] The token cache is being invalidate because project 9edab973767545988e178ddff76557ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:19:46.354280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.354905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.356455 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1331/2630] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:19:46 2022] DELETE /identity/v3/projects/9edab973767545988e178ddff76557ac => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:19:46.360497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:19:46.361969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:19:46.375820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04917023-4e2f-44ff-aedf-e5aa07ca332d admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:19:46.430259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:19:46.431133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:19:46.431413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:19:46.431689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] PATH_INFO: `/v3/users/80c376e37e9041e79ac92e72f4d71468` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:19:46.432901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] RBAC: Authorizing `identity:delete_user(user_id=80c376e37e9041e79ac92e72f4d71468)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:19:46.434835 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:19:46.479461 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80c376e37e9041e79ac92e72f4d71468'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.486105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80c376e37e9041e79ac92e72f4d71468'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.486757 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80c376e37e9041e79ac92e72f4d71468'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.487383 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80c376e37e9041e79ac92e72f4d71468'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.499143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80c376e37e9041e79ac92e72f4d71468'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.501376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:19:46.513206 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1300/2631] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:19:46 2022] DELETE /identity/v3/users/80c376e37e9041e79ac92e72f4d71468 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:19:46.679595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:19:46.681100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:19:46.694266 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631d5dce-2100-4f5c-b5b3-a2cf4b5d1bed admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:19:46.750178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:19:46.751577 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:19:46.751860 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:19:46.752138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] PATH_INFO: `/v3/projects/21761824bf0a46f996a66abd2b210e6d` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:19:46.753385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21761824bf0a46f996a66abd2b210e6d)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:19:46.755843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:19:46.805740 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21761824bf0a46f996a66abd2b210e6d', 'request_id': 'req-2b834333-81d0-47d4-a38f-a96f8e023224'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.811865 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21761824bf0a46f996a66abd2b210e6d', 'request_id': 'req-2b834333-81d0-47d4-a38f-a96f8e023224'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.820750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21761824bf0a46f996a66abd2b210e6d', 'request_id': 'req-2b834333-81d0-47d4-a38f-a96f8e023224'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.832506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] The token cache is being invalidate because project 21761824bf0a46f996a66abd2b210e6d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:19:46.832791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.833413 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:19:46.834936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1332/2632] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:19:46 2022] DELETE /identity/v3/projects/21761824bf0a46f996a66abd2b210e6d => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:19:56.512453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:19:56.829537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=57,keystone:DELETE=9,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:20:06.655524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:06.656953 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:06.713543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc535ce5-9e72-49c6-911b-19fd473ff90b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:06.761477 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd073d9-ed2d-4abe-a852-fa646d55a2fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:06.762117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd073d9-ed2d-4abe-a852-fa646d55a2fd service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:06.762404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd073d9-ed2d-4abe-a852-fa646d55a2fd service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:06.762677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd073d9-ed2d-4abe-a852-fa646d55a2fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:06.764443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ecd073d9-ed2d-4abe-a852-fa646d55a2fd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:06.834642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd073d9-ed2d-4abe-a852-fa646d55a2fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:06.835989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd073d9-ed2d-4abe-a852-fa646d55a2fd service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:06.847253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1301/2633] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:20:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:20:08.018746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:08.020246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:08.033868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b834333-81d0-47d4-a38f-a96f8e023224 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:08.037429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:08.038407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:08.038696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:08.038975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:08.040725 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:08.054548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:08.055964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:08.066838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1333/2634] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:20:08 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:20:16.841922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=74 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:20:18.061957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=6 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:20:19.462299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e2312a-d548-40cb-9331-0b3c21f57c1c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:19.462760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e2312a-d548-40cb-9331-0b3c21f57c1c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:19.462892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e2312a-d548-40cb-9331-0b3c21f57c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:19.481292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-a9e2312a-d548-40cb-9331-0b3c21f57c1c None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:20:19.482501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a9e2312a-d548-40cb-9331-0b3c21f57c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:19.537126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1302/2635] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:20:19 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:20:19.581979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:19.583385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:19.651718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00a2f15-ed56-4ab7-a6d1-b8381e2b83ab service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:19.657691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c05f1f6-493d-45ae-9c08-5d4e794b1f5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:19.658345 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c05f1f6-493d-45ae-9c08-5d4e794b1f5d service swift] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:19.658629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c05f1f6-493d-45ae-9c08-5d4e794b1f5d service swift] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:19.658905 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c05f1f6-493d-45ae-9c08-5d4e794b1f5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:19.667878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c05f1f6-493d-45ae-9c08-5d4e794b1f5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:19.669504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c05f1f6-493d-45ae-9c08-5d4e794b1f5d service swift] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:19.677965 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1334/2636] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:20:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:20:29.530073 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=25 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:20:29.674201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=32 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:20:30.213823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e2312a-d548-40cb-9331-0b3c21f57c1c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:30.223274 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e2312a-d548-40cb-9331-0b3c21f57c1c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:30.227784 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4285399c-b865-474a-88c8-32d2a3bf0ba3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:30.228485 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285399c-b865-474a-88c8-32d2a3bf0ba3 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:30.228774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285399c-b865-474a-88c8-32d2a3bf0ba3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:30.229053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4285399c-b865-474a-88c8-32d2a3bf0ba3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:30.230850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4285399c-b865-474a-88c8-32d2a3bf0ba3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:30.240328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285399c-b865-474a-88c8-32d2a3bf0ba3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:30.242302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4285399c-b865-474a-88c8-32d2a3bf0ba3 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:30.250536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1303/2637] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:20:30 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:20:33.720020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c74c34-208d-452c-9715-338d0e4c5950 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:33.720597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c74c34-208d-452c-9715-338d0e4c5950 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:33.720597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c74c34-208d-452c-9715-338d0e4c5950 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:33.722139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1335/2638] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:20:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:20:33.728904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7121a9-e265-44d1-b6cf-ab015552a91d None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:33.729376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7121a9-e265-44d1-b6cf-ab015552a91d None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:33.729469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7121a9-e265-44d1-b6cf-ab015552a91d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:33.749290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8c7121a9-e265-44d1-b6cf-ab015552a91d None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:20:33.750503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8c7121a9-e265-44d1-b6cf-ab015552a91d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:33.759828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1304/2639] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:20:33 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:20:33.771701 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c74c34-208d-452c-9715-338d0e4c5950 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:33.773468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b9c74c34-208d-452c-9715-338d0e4c5950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:33.837377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c74c34-208d-452c-9715-338d0e4c5950 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:33.841263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e83d928-4ebd-447f-919e-097d6ee6846d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:33.841931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e83d928-4ebd-447f-919e-097d6ee6846d service glance] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:33.842214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e83d928-4ebd-447f-919e-097d6ee6846d service glance] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:33.842493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e83d928-4ebd-447f-919e-097d6ee6846d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:33.854857 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e83d928-4ebd-447f-919e-097d6ee6846d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:33.856319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e83d928-4ebd-447f-919e-097d6ee6846d service glance] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:33.867793 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1336/2640] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:20:33 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:20:33.933758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212dd728-bbd8-4528-98cb-a1430d390566 None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:33.934069 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212dd728-bbd8-4528-98cb-a1430d390566 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:33.934344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212dd728-bbd8-4528-98cb-a1430d390566 None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:33.935724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1305/2641] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:20:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:20:33.940576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15059c58-18e3-47e7-b7b8-5700f33f68c3 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:33.940862 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15059c58-18e3-47e7-b7b8-5700f33f68c3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:33.941136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15059c58-18e3-47e7-b7b8-5700f33f68c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:33.977508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-15059c58-18e3-47e7-b7b8-5700f33f68c3 None None] MFA Rules not processed for user `36ce9bd8ce6448b8b09fe9eb7b8f7c54`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:20:33.978624 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-15059c58-18e3-47e7-b7b8-5700f33f68c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:33.985041 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1337/2642] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:20:33 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:20:33.996833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212dd728-bbd8-4528-98cb-a1430d390566 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:33.998495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-212dd728-bbd8-4528-98cb-a1430d390566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:34.011390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212dd728-bbd8-4528-98cb-a1430d390566 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:34.014949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82387d9d-f3cb-4876-8213-309c53fe150e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:34.015624 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82387d9d-f3cb-4876-8213-309c53fe150e service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:34.015912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82387d9d-f3cb-4876-8213-309c53fe150e service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:34.016220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82387d9d-f3cb-4876-8213-309c53fe150e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:34.023909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82387d9d-f3cb-4876-8213-309c53fe150e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:34.025358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82387d9d-f3cb-4876-8213-309c53fe150e service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:34.033092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1306/2643] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:20:33 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:20:34.116867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1937a99e-4aab-4266-baf1-8e5ed6f07109 None None] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:34.117420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1937a99e-4aab-4266-baf1-8e5ed6f07109 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:34.117695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1937a99e-4aab-4266-baf1-8e5ed6f07109 None None] PATH_INFO: `/` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:34.119094 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1338/2644] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:20:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:20:34.124815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb057d99-3027-4135-b391-96f0c85da4ec None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:34.125101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb057d99-3027-4135-b391-96f0c85da4ec None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:34.125377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb057d99-3027-4135-b391-96f0c85da4ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:34.143024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fb057d99-3027-4135-b391-96f0c85da4ec None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:20:34.144232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fb057d99-3027-4135-b391-96f0c85da4ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:34.153922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1307/2645] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:20:34 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:20:34.164770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1937a99e-4aab-4266-baf1-8e5ed6f07109 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:34.175767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1937a99e-4aab-4266-baf1-8e5ed6f07109 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:34.179347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:34.179986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:34.180270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:34.180587 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:34.190619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:34.192047 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:34.203207 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1339/2646] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:20:34 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:20:36.557027 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:36.557506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:36.557616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:36.575223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:20:36.576668 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:36.583774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1308/2647] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:20:36 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:20:36.595891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:36.606931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f7766e-d5fc-46d3-b795-4a4935e063a1 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:36.611767 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:36.612409 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:36.612708 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:36.612983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:36.621156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:36.622719 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:36.630608 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1340/2648] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:20:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 05 10:20:46.569825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=21 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:20:46.627367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=47 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:20:47.746471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:20:47.747998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:47.763834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd5cd6c-f090-496e-979e-ca8bd2ba1ebe None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:20:47.769208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83137e01-9f2d-4b14-a801-3cb1c0281501 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:20:47.769846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83137e01-9f2d-4b14-a801-3cb1c0281501 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:20:47.770129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83137e01-9f2d-4b14-a801-3cb1c0281501 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:20:47.770405 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83137e01-9f2d-4b14-a801-3cb1c0281501 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:20:47.772089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-83137e01-9f2d-4b14-a801-3cb1c0281501 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:20:47.851481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83137e01-9f2d-4b14-a801-3cb1c0281501 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:20:47.853127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83137e01-9f2d-4b14-a801-3cb1c0281501 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:20:47.871433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1309/2649] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:20:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:20:57.868438 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=41 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:02.512414 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:02.513849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:02.529923 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a969fac-156c-4432-a1c7-901dac98b9a5 service swift] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:02.538126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:02.538893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:02.539198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:02.539523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:02.542333 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:02.567960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:02.569809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:02.588615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1341/2650] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:02 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:03.076125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9124a99f-bdb7-490e-bb2d-f2d1829f96a9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:03.076610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9124a99f-bdb7-490e-bb2d-f2d1829f96a9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:03.076694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9124a99f-bdb7-490e-bb2d-f2d1829f96a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:03.106006 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-9124a99f-bdb7-490e-bb2d-f2d1829f96a9 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:03.107259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9124a99f-bdb7-490e-bb2d-f2d1829f96a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:03.124677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1310/2651] 10.222.0.20 () {58 vars in 967 bytes} [Tue Jul 5 10:21:03 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:03.135196 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:03.147849 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb981abf-488a-4320-a899-141d5d80fae8 service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:03.153142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:03.153924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:03.154210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:03.154483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:03.164161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:03.165792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:03.174128 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1342/2652] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:21:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:21:08.763521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9124a99f-bdb7-490e-bb2d-f2d1829f96a9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:08.773122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9124a99f-bdb7-490e-bb2d-f2d1829f96a9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:08.777276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:08.777922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:08.778205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:08.778483 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:08.780243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:08.843950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:08.845415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:08.855135 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1311/2653] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:11.689801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:11.691337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:11.706418 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69a2f2f-e54b-4a20-a758-4d67cd38422f service swift] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:11.711210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:11.712146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:11.712548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:11.712830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] PATH_INFO: `/v3/users/036cb887c7894e75ba3eb4c1a8b21c69` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:11.714613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] RBAC: Authorizing `identity:delete_user(user_id=036cb887c7894e75ba3eb4c1a8b21c69)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:11.716601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:11.768062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '036cb887c7894e75ba3eb4c1a8b21c69'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:11.768824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '036cb887c7894e75ba3eb4c1a8b21c69'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:11.775938 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '036cb887c7894e75ba3eb4c1a8b21c69'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:11.776603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '036cb887c7894e75ba3eb4c1a8b21c69'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:11.783979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-c76d455e-dd08-4084-89f7-bad58eab6a86 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '036cb887c7894e75ba3eb4c1a8b21c69'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:11.803269 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1343/2654] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:11 2022] DELETE /identity/v3/users/036cb887c7894e75ba3eb4c1a8b21c69 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:21:11.977405 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:11.991166 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c265e97-846f-4c5f-8509-50bb1b2ef55c service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.049341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.050203 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.050488 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.050764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] PATH_INFO: `/v3/projects/7eccead420ae4aacb6f31dcb17a138cc` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.051913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eccead420ae4aacb6f31dcb17a138cc)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.054105 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.100039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eccead420ae4aacb6f31dcb17a138cc', 'request_id': 'req-389014a2-b946-4127-bb26-7b59700e5c64'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:12.105978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eccead420ae4aacb6f31dcb17a138cc', 'request_id': 'req-389014a2-b946-4127-bb26-7b59700e5c64'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:12.112071 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eccead420ae4aacb6f31dcb17a138cc', 'request_id': 'req-389014a2-b946-4127-bb26-7b59700e5c64'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:12.124303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] The token cache is being invalidate because project 7eccead420ae4aacb6f31dcb17a138cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:12.124646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:12.125294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-389014a2-b946-4127-bb26-7b59700e5c64 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:12.126837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1312/2655] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:11 2022] DELETE /identity/v3/projects/7eccead420ae4aacb6f31dcb17a138cc => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:12.131926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c20179-1d5f-410a-ad44-6c6087d56117 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.132213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c20179-1d5f-410a-ad44-6c6087d56117 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.132523 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c20179-1d5f-410a-ad44-6c6087d56117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.150246 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d3c20179-1d5f-410a-ad44-6c6087d56117 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:12.151935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d3c20179-1d5f-410a-ad44-6c6087d56117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:12.210096 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1344/2656] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:12.218893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb04c50a-b03c-45f0-a81f-d870edf05769 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.219183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb04c50a-b03c-45f0-a81f-d870edf05769 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.219464 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb04c50a-b03c-45f0-a81f-d870edf05769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.238129 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fb04c50a-b03c-45f0-a81f-d870edf05769 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:12.239472 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fb04c50a-b03c-45f0-a81f-d870edf05769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:12.247713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1313/2657] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:12.251684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c20179-1d5f-410a-ad44-6c6087d56117 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.261421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c20179-1d5f-410a-ad44-6c6087d56117 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.266010 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.266691 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.266978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.267259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.267990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:21:12.268297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.269932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.281795 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1345/2658] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:21:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:21:12.286014 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb04c50a-b03c-45f0-a81f-d870edf05769 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.295679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb04c50a-b03c-45f0-a81f-d870edf05769 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.300242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.300917 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.301202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.301480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.302155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.304030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.311989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:12.331303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1314/2659] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:12 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:12.335480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.345958 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ba20c-10b0-4808-9d8b-848a865b44e4 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.376566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:12.405143 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.405807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.406090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.406368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.407053 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.409030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.427162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1346/2660] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:12 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:12.431316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.441418 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b91d12a-63d8-45b6-88d4-db4e2f69e199 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.445901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.446581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.446866 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.447141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.447861 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.449975 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.459983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1315/2661] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:12.463644 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.474007 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7a8f8-e434-4681-91d7-5cae99ef1706 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.478665 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a7c3fc-18c6-414d-b8ef-2246fc4aae84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.479312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7c3fc-18c6-414d-b8ef-2246fc4aae84 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.479596 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7c3fc-18c6-414d-b8ef-2246fc4aae84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.479919 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7c3fc-18c6-414d-b8ef-2246fc4aae84 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.480669 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a7c3fc-18c6-414d-b8ef-2246fc4aae84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.482261 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a7c3fc-18c6-414d-b8ef-2246fc4aae84 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.492020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1347/2662] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:12.496246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.506104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c6c66b-a4a2-4b8e-a689-4b01fcc696fc admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.511334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.512226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.512550 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.512828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] PATH_INFO: `/v3/projects/c649d9d862e54d55a4d901ef8d15abf9/users/5bf890db7def45738a614086056bc89d/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.528025 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c649d9d862e54d55a4d901ef8d15abf9, user_id=5bf890db7def45738a614086056bc89d, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.530719 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.543316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1316/2663] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:12 2022] PUT /identity/v3/projects/c649d9d862e54d55a4d901ef8d15abf9/users/5bf890db7def45738a614086056bc89d/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:12.554167 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97326459-ecef-4a90-b99a-7b625d7481b6 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.554461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97326459-ecef-4a90-b99a-7b625d7481b6 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.554743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97326459-ecef-4a90-b99a-7b625d7481b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.581488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-97326459-ecef-4a90-b99a-7b625d7481b6 None None] MFA Rules not processed for user `5bf890db7def45738a614086056bc89d`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:12.582715 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-97326459-ecef-4a90-b99a-7b625d7481b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:12.646125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1348/2664] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:12 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:12.650531 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.711898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842102f8-db80-4e17-9cfb-727d4192d141 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.717035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.717978 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.718256 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.718946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.720909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.746367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1317/2665] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:12 2022] POST /identity/v3/projects => generated 371 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:12.750472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97326459-ecef-4a90-b99a-7b625d7481b6 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.761195 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97326459-ecef-4a90-b99a-7b625d7481b6 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.818240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.818925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.819214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.819499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.820185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.822175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.841784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1349/2666] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:12 2022] POST /identity/v3/users => generated 292 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:12.845701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.856554 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8912d1-223f-4888-b067-6a907ec3c37f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.861341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.862001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.862290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.862571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.863329 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.864972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.875650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1318/2667] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:12.879493 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.889337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a012c7-552a-418d-86bc-32cf12e30b96 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.893966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.894866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.895151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.895428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] PATH_INFO: `/v3/projects/c8b853c4711c482e8830f18ee7f1b2dd/users/2477e74f49984cc78d51658fe065bb58/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.910699 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8b853c4711c482e8830f18ee7f1b2dd, user_id=2477e74f49984cc78d51658fe065bb58, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:12.913349 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:12.926161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1350/2668] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:12 2022] PUT /identity/v3/projects/c8b853c4711c482e8830f18ee7f1b2dd/users/2477e74f49984cc78d51658fe065bb58/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:12.930136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:12.940613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ebc822-1a70-4a5e-ae6f-35c4a1a05438 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:12.996679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57db48f5-e993-4ca5-8c8f-ef6ff74228ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:12.997338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57db48f5-e993-4ca5-8c8f-ef6ff74228ee admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:12.997628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57db48f5-e993-4ca5-8c8f-ef6ff74228ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:12.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57db48f5-e993-4ca5-8c8f-ef6ff74228ee admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:12.998638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57db48f5-e993-4ca5-8c8f-ef6ff74228ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:13.000222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57db48f5-e993-4ca5-8c8f-ef6ff74228ee admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:13.009870 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1319/2669] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:12 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:13.014775 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:13.025123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e595a2-2196-4b18-821b-e02eae71b97d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:13.030514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:13.031399 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:13.031685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:13.031971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] PATH_INFO: `/v3/projects/c8b853c4711c482e8830f18ee7f1b2dd/users/2477e74f49984cc78d51658fe065bb58/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:13.033943 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8b853c4711c482e8830f18ee7f1b2dd, user_id=2477e74f49984cc78d51658fe065bb58, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:13.036806 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:13.039463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:13.050363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1351/2670] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:13 2022] PUT /identity/v3/projects/c8b853c4711c482e8830f18ee7f1b2dd/users/2477e74f49984cc78d51658fe065bb58/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:13.060645 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b4052-bd69-490f-a6b6-4f003e77625e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:13.060945 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b4052-bd69-490f-a6b6-4f003e77625e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:13.061237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b4052-bd69-490f-a6b6-4f003e77625e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:13.086886 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-096b4052-bd69-490f-a6b6-4f003e77625e None None] MFA Rules not processed for user `2477e74f49984cc78d51658fe065bb58`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:13.088138 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-096b4052-bd69-490f-a6b6-4f003e77625e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:13.100647 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:13.157788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1320/2671] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:13 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:13.169641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:13.171347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:13.227037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8a79f1-deb2-4ca7-ad1d-35662d3e9c20 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:13.231156 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30cfb6b-ee37-4736-877c-00186c925181 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:13.231788 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30cfb6b-ee37-4736-877c-00186c925181 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:13.232067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30cfb6b-ee37-4736-877c-00186c925181 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:13.232340 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30cfb6b-ee37-4736-877c-00186c925181 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:13.241177 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30cfb6b-ee37-4736-877c-00186c925181 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:13.242592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30cfb6b-ee37-4736-877c-00186c925181 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:13.252757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1352/2672] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:13 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:13.614539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096b4052-bd69-490f-a6b6-4f003e77625e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:13.677726 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096b4052-bd69-490f-a6b6-4f003e77625e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:13.682169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:13.683065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:13.683345 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:13.683621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] PATH_INFO: `/v3/users/5bf890db7def45738a614086056bc89d` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:13.684840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bf890db7def45738a614086056bc89d)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:13.686766 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:13.731616 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bf890db7def45738a614086056bc89d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:13.738463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bf890db7def45738a614086056bc89d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:13.739108 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bf890db7def45738a614086056bc89d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:13.739710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bf890db7def45738a614086056bc89d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:13.751481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bf890db7def45738a614086056bc89d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:13.763878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1321/2673] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:13 2022] DELETE /identity/v3/users/5bf890db7def45738a614086056bc89d => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:13.768319 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:13.768836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:13.768986 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:13.786251 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:13.787479 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:13.845595 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1353/2674] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:13.855652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:13.857158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:13.889032 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:13.916401 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2effbc2d-ac5d-4ee8-8f73-366b94f804a1 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:13.920072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:13.920759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:13.921047 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:13.921324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:13.930759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:13.932200 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:13.942317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1322/2675] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:14.208267 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:14.209772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:14.227673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bbbb54-3ae1-402a-9cd2-32a0118befd5 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:14.234185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:14.235073 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:14.235337 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:14.235619 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] PATH_INFO: `/v3/projects/c649d9d862e54d55a4d901ef8d15abf9` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:14.236853 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c649d9d862e54d55a4d901ef8d15abf9)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:14.238799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:14.285605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c649d9d862e54d55a4d901ef8d15abf9', 'request_id': 'req-27e0bccd-a20b-4c1a-864c-83f569421e3c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.291306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c649d9d862e54d55a4d901ef8d15abf9', 'request_id': 'req-27e0bccd-a20b-4c1a-864c-83f569421e3c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.299139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c649d9d862e54d55a4d901ef8d15abf9', 'request_id': 'req-27e0bccd-a20b-4c1a-864c-83f569421e3c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.310397 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] The token cache is being invalidate because project c649d9d862e54d55a4d901ef8d15abf9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:14.310689 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.311321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.312909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1354/2676] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:14 2022] DELETE /identity/v3/projects/c649d9d862e54d55a4d901ef8d15abf9 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:14.316898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:14.326415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be669158-0ab8-4ea4-8564-38a28a171fb8 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:14.382263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:14.383133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:14.383424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:14.383704 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] PATH_INFO: `/v3/users/2477e74f49984cc78d51658fe065bb58` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:14.384923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2477e74f49984cc78d51658fe065bb58)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:14.387408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:14.431041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2477e74f49984cc78d51658fe065bb58'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.438315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2477e74f49984cc78d51658fe065bb58'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.438966 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2477e74f49984cc78d51658fe065bb58'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.439579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2477e74f49984cc78d51658fe065bb58'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.451853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55d8397b-b432-4f9b-a9f7-fe0563586165 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2477e74f49984cc78d51658fe065bb58'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.463962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1323/2677] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:14 2022] DELETE /identity/v3/users/2477e74f49984cc78d51658fe065bb58 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:14.730918 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:14.732374 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:14.745540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e0bccd-a20b-4c1a-864c-83f569421e3c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:14.764052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:14.806733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:14.807637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:14.807926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:14.808201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] PATH_INFO: `/v3/projects/c8b853c4711c482e8830f18ee7f1b2dd` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:14.809514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8b853c4711c482e8830f18ee7f1b2dd)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:14.811420 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:14.856670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8b853c4711c482e8830f18ee7f1b2dd', 'request_id': 'req-843697f1-f4bd-4393-94d9-5e9449ccf8d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.862018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8b853c4711c482e8830f18ee7f1b2dd', 'request_id': 'req-843697f1-f4bd-4393-94d9-5e9449ccf8d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.868961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8b853c4711c482e8830f18ee7f1b2dd', 'request_id': 'req-843697f1-f4bd-4393-94d9-5e9449ccf8d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.880305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] The token cache is being invalidate because project c8b853c4711c482e8830f18ee7f1b2dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:14.880620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.881237 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-843697f1-f4bd-4393-94d9-5e9449ccf8d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:14.882736 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1355/2678] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:14 2022] DELETE /identity/v3/projects/c8b853c4711c482e8830f18ee7f1b2dd => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:14.888149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29993fb7-6b96-431b-962c-c7e4c60f2fa4 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:14.888505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29993fb7-6b96-431b-962c-c7e4c60f2fa4 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:14.888790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29993fb7-6b96-431b-962c-c7e4c60f2fa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:14.907106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-29993fb7-6b96-431b-962c-c7e4c60f2fa4 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:14.908305 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-29993fb7-6b96-431b-962c-c7e4c60f2fa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:14.954228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:14.972085 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1324/2679] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:14.981162 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fdd0e3-d4c1-4c23-88e2-c3c4fe63cf93 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:14.981387 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fdd0e3-d4c1-4c23-88e2-c3c4fe63cf93 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:14.981662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fdd0e3-d4c1-4c23-88e2-c3c4fe63cf93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.000064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c9fdd0e3-d4c1-4c23-88e2-c3c4fe63cf93 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:15.001282 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c9fdd0e3-d4c1-4c23-88e2-c3c4fe63cf93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:15.009110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1356/2680] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:15.014243 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29993fb7-6b96-431b-962c-c7e4c60f2fa4 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.024098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29993fb7-6b96-431b-962c-c7e4c60f2fa4 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.029208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.029894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.030182 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.030508 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.031253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:21:15.031567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.033219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.045197 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1325/2681] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:21:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:21:15.049550 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fdd0e3-d4c1-4c23-88e2-c3c4fe63cf93 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.059359 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fdd0e3-d4c1-4c23-88e2-c3c4fe63cf93 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.064336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.065019 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.065301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.065576 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.066263 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.068175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.094173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1357/2682] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:15 2022] POST /identity/v3/projects => generated 365 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:15.098462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.109213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52d5944-8839-4985-bb06-bd2a6dfa2b26 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.164713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.165370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.165650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.165924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.166605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.169216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.189850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1326/2683] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:15 2022] POST /identity/v3/users => generated 283 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:15.193937 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.204776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a651d7a1-3689-4e16-ab0a-7fe5eb443170 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.209421 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.210080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.210365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.210647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.211372 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.213009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.223537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1358/2684] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:15.227589 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.237845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04692793-e7d7-4221-a550-a6b35cd3bdd5 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.242371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db58f59-ac23-4276-9379-ab8a7b7dfe08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.243059 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db58f59-ac23-4276-9379-ab8a7b7dfe08 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.243344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db58f59-ac23-4276-9379-ab8a7b7dfe08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.243623 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db58f59-ac23-4276-9379-ab8a7b7dfe08 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.244347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db58f59-ac23-4276-9379-ab8a7b7dfe08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.246573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db58f59-ac23-4276-9379-ab8a7b7dfe08 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.256302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1327/2685] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:15.260210 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.270489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd0a6a-7330-4fea-943d-4c04e48e6aa0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.275288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.276175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.276507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.276790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] PATH_INFO: `/v3/projects/4a430c39722b443ba595dfbb3a9c907d/users/b657bf1b5f5844f4a85d2beb1ca615bd/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.291175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a430c39722b443ba595dfbb3a9c907d, user_id=b657bf1b5f5844f4a85d2beb1ca615bd, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.293887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.306403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1359/2686] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:15 2022] PUT /identity/v3/projects/4a430c39722b443ba595dfbb3a9c907d/users/b657bf1b5f5844f4a85d2beb1ca615bd/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:15.317131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723be6d5-e028-4e8b-9d8a-aa63f5af7593 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.317629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723be6d5-e028-4e8b-9d8a-aa63f5af7593 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.317730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723be6d5-e028-4e8b-9d8a-aa63f5af7593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.343862 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-723be6d5-e028-4e8b-9d8a-aa63f5af7593 None None] MFA Rules not processed for user `b657bf1b5f5844f4a85d2beb1ca615bd`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:15.345094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-723be6d5-e028-4e8b-9d8a-aa63f5af7593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:15.406999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1328/2687] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:15 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:15.411424 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.459731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:15.474794 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-370b5593-67f1-4be4-a426-fa30d1d604d9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.479453 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.480634 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.480926 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.481208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.481901 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.483803 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.512290 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1360/2688] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:15 2022] POST /identity/v3/projects => generated 365 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:15.513364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723be6d5-e028-4e8b-9d8a-aa63f5af7593 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.524202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723be6d5-e028-4e8b-9d8a-aa63f5af7593 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.567396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:15.580449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.581121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.581424 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.581686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.582376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.584277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.603440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1329/2689] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:15 2022] POST /identity/v3/users => generated 289 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:15.607932 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.618307 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cee60c-eebf-4fb0-9569-6363b7f183ed admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.623866 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.624567 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.624859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.625141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.625873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.627499 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.637875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1361/2690] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:15.641965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.651552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c426ef-e7c0-4a32-b077-20e2c1aab86e admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.656080 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.656985 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.657264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.657539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] PATH_INFO: `/v3/projects/3223df1b08eb4f70ac2508fdf49be9b2/users/058fa44816484f5898552fab35c716df/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.672065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] RBAC: Authorizing `identity:create_grant(project_id=3223df1b08eb4f70ac2508fdf49be9b2, user_id=058fa44816484f5898552fab35c716df, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.674731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.687910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1330/2691] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:15 2022] PUT /identity/v3/projects/3223df1b08eb4f70ac2508fdf49be9b2/users/058fa44816484f5898552fab35c716df/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:15.691824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.702344 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9585c989-67d3-4473-a07e-2a31e0d1b142 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.762002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257d8315-65e9-4b58-bad4-77fc2b45dcf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.762733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257d8315-65e9-4b58-bad4-77fc2b45dcf1 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.763013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257d8315-65e9-4b58-bad4-77fc2b45dcf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.763320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257d8315-65e9-4b58-bad4-77fc2b45dcf1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.764044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257d8315-65e9-4b58-bad4-77fc2b45dcf1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.765670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257d8315-65e9-4b58-bad4-77fc2b45dcf1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.775757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1362/2692] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:15 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:15.780845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.791649 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ba7786-1419-42ec-bca6-ef545038361c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:15.797901 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:15.798999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.799603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.800053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] PATH_INFO: `/v3/projects/3223df1b08eb4f70ac2508fdf49be9b2/users/058fa44816484f5898552fab35c716df/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.802205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3223df1b08eb4f70ac2508fdf49be9b2, user_id=058fa44816484f5898552fab35c716df, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:15.805109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:15.817328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1331/2693] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:15 2022] PUT /identity/v3/projects/3223df1b08eb4f70ac2508fdf49be9b2/users/058fa44816484f5898552fab35c716df/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:15.828240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7026fe1d-f50b-4ea9-971d-ba9bbfc8aad3 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:15.828750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7026fe1d-f50b-4ea9-971d-ba9bbfc8aad3 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:15.828842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7026fe1d-f50b-4ea9-971d-ba9bbfc8aad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:15.868843 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-7026fe1d-f50b-4ea9-971d-ba9bbfc8aad3 None None] MFA Rules not processed for user `058fa44816484f5898552fab35c716df`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:15.870085 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7026fe1d-f50b-4ea9-971d-ba9bbfc8aad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:15.945834 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1363/2694] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:15 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:21:15.956577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:15.958297 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:16.021871 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8251e446-4eeb-4105-b69b-e80f936a7db9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:16.026303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:16.027644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:16.027936 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:16.028221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:16.039435 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:16.041019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:16.051259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1332/2695] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:15 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:16.122914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7026fe1d-f50b-4ea9-971d-ba9bbfc8aad3 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:16.132676 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7026fe1d-f50b-4ea9-971d-ba9bbfc8aad3 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:16.136836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af169b20-0a86-4872-944c-af9fba052e33 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:16.137474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af169b20-0a86-4872-944c-af9fba052e33 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:16.137762 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af169b20-0a86-4872-944c-af9fba052e33 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:16.138045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af169b20-0a86-4872-944c-af9fba052e33 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:16.185606 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:16.203996 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af169b20-0a86-4872-944c-af9fba052e33 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:16.205494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af169b20-0a86-4872-944c-af9fba052e33 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:16.214980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1364/2696] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:16 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:18.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:18.951264 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc59f3d8-a73b-480e-99d3-7b81818c99d6 service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:18.956578 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:18.957533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:18.957825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:18.958111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] PATH_INFO: `/v3/users/b657bf1b5f5844f4a85d2beb1ca615bd` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:18.959328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=b657bf1b5f5844f4a85d2beb1ca615bd)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:18.961373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:19.006202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:19.009050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b657bf1b5f5844f4a85d2beb1ca615bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.018024 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b657bf1b5f5844f4a85d2beb1ca615bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.018711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b657bf1b5f5844f4a85d2beb1ca615bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.019295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b657bf1b5f5844f4a85d2beb1ca615bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.031507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b657bf1b5f5844f4a85d2beb1ca615bd'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.044340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1333/2697] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:18 2022] DELETE /identity/v3/users/b657bf1b5f5844f4a85d2beb1ca615bd => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:19.048854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:19.049342 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:19.049460 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:19.066742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:19.067945 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:19.125838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1365/2698] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:19.135822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:19.137232 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:19.192944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b4cf3c-7c64-4aa8-9fb1-983efa54f0fd admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:19.196577 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:19.197218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:19.198018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:19.198292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:19.207156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:19.208631 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:19.218606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1334/2699] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:19.478214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:19.480253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:19.492667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3913546-2276-42bd-ba19-db47b677db2d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:19.497761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:19.498623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:19.498904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:19.499178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] PATH_INFO: `/v3/projects/4a430c39722b443ba595dfbb3a9c907d` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:19.500360 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a430c39722b443ba595dfbb3a9c907d)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:19.502406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:19.548755 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a430c39722b443ba595dfbb3a9c907d', 'request_id': 'req-35997a96-5589-4a75-a98a-20285151ecb8'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.554328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a430c39722b443ba595dfbb3a9c907d', 'request_id': 'req-35997a96-5589-4a75-a98a-20285151ecb8'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.561432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a430c39722b443ba595dfbb3a9c907d', 'request_id': 'req-35997a96-5589-4a75-a98a-20285151ecb8'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.572658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] The token cache is being invalidate because project 4a430c39722b443ba595dfbb3a9c907d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:19.572941 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.573558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.575086 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1366/2700] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:19 2022] DELETE /identity/v3/projects/4a430c39722b443ba595dfbb3a9c907d => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:19.579154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:19.588749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e53752-ca82-4de8-8cde-b567011ef395 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:19.645180 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:19.646050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:19.646335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:19.646613 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] PATH_INFO: `/v3/users/058fa44816484f5898552fab35c716df` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:19.647794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] RBAC: Authorizing `identity:delete_user(user_id=058fa44816484f5898552fab35c716df)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:19.649764 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:19.694641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '058fa44816484f5898552fab35c716df'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.701497 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '058fa44816484f5898552fab35c716df'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.702150 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '058fa44816484f5898552fab35c716df'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.702782 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '058fa44816484f5898552fab35c716df'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.709725 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:19.716174 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-156c294f-e185-4dd3-96a6-238fb350e728 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '058fa44816484f5898552fab35c716df'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:19.729357 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1335/2701] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:19 2022] DELETE /identity/v3/users/058fa44816484f5898552fab35c716df => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:19.999659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.001098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:20.017119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35997a96-5589-4a75-a98a-20285151ecb8 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.076119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.077021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.077309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.077657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] PATH_INFO: `/v3/projects/3223df1b08eb4f70ac2508fdf49be9b2` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.078812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3223df1b08eb4f70ac2508fdf49be9b2)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.080770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.086961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:20.128633 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3223df1b08eb4f70ac2508fdf49be9b2', 'request_id': 'req-e2d7d564-bafa-4d64-b934-f8d92082b018'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:20.134111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3223df1b08eb4f70ac2508fdf49be9b2', 'request_id': 'req-e2d7d564-bafa-4d64-b934-f8d92082b018'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:20.141312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3223df1b08eb4f70ac2508fdf49be9b2', 'request_id': 'req-e2d7d564-bafa-4d64-b934-f8d92082b018'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:20.152874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] The token cache is being invalidate because project 3223df1b08eb4f70ac2508fdf49be9b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:20.153165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:20.153778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e2d7d564-bafa-4d64-b934-f8d92082b018 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:20.155283 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1367/2702] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:19 2022] DELETE /identity/v3/projects/3223df1b08eb4f70ac2508fdf49be9b2 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:20.160140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f0a72f-2d52-42c7-925f-4be587495fe0 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.160629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f0a72f-2d52-42c7-925f-4be587495fe0 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.160773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f0a72f-2d52-42c7-925f-4be587495fe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.180096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-87f0a72f-2d52-42c7-925f-4be587495fe0 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:20.181338 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-87f0a72f-2d52-42c7-925f-4be587495fe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:20.241692 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1336/2703] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:20.250735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92a5043-26b4-48f6-a4f8-4265278ac58c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.251022 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92a5043-26b4-48f6-a4f8-4265278ac58c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.251300 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92a5043-26b4-48f6-a4f8-4265278ac58c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.269221 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-d92a5043-26b4-48f6-a4f8-4265278ac58c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:20.270406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-d92a5043-26b4-48f6-a4f8-4265278ac58c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:20.278617 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1368/2704] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:20.282763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f0a72f-2d52-42c7-925f-4be587495fe0 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.293002 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f0a72f-2d52-42c7-925f-4be587495fe0 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.297608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.298258 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.298541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.298819 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.299540 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:21:20.299849 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.301489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.313056 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1337/2705] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:21:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:21:20.316989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92a5043-26b4-48f6-a4f8-4265278ac58c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.326170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92a5043-26b4-48f6-a4f8-4265278ac58c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.331295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.331882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.332166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.332481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.333180 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.335089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.361095 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1369/2706] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:20 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:21:20.365373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.375469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e099cf-ebca-45d0-8047-9f73fe8581f4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.430548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.431215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.431498 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.431776 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.432487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.434427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.453087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1338/2707] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:20 2022] POST /identity/v3/users => generated 293 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:20.457264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.468001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19864136-70e3-4773-8964-88ae185afba3 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.472659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.473315 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.473598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.473878 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.474622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.476242 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.488093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1370/2708] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:20.491796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.503491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e13a0c9-b2e4-47ff-88c2-abea34ebfe10 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.508799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd323ff7-3410-449e-8e4a-5245d6512a46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.509453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd323ff7-3410-449e-8e4a-5245d6512a46 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.509741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd323ff7-3410-449e-8e4a-5245d6512a46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.510020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd323ff7-3410-449e-8e4a-5245d6512a46 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.511347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd323ff7-3410-449e-8e4a-5245d6512a46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.512963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd323ff7-3410-449e-8e4a-5245d6512a46 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.522824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1339/2709] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:20.526830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.536940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb9a62-bf31-4af8-bb8c-36a233cc51af admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.541776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.542657 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.542939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.543214 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] PATH_INFO: `/v3/projects/619488d546174e1cb3a7bfe566eef450/users/e6563259190c47b88baf33c10d76ffde/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.558552 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] RBAC: Authorizing `identity:create_grant(project_id=619488d546174e1cb3a7bfe566eef450, user_id=e6563259190c47b88baf33c10d76ffde, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.561254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.574100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1371/2710] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:20 2022] PUT /identity/v3/projects/619488d546174e1cb3a7bfe566eef450/users/e6563259190c47b88baf33c10d76ffde/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:20.584267 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683148a-adb4-48e5-8382-961472075bc1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.584542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683148a-adb4-48e5-8382-961472075bc1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.584830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1683148a-adb4-48e5-8382-961472075bc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.610912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-1683148a-adb4-48e5-8382-961472075bc1 None None] MFA Rules not processed for user `e6563259190c47b88baf33c10d76ffde`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:20.612124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-1683148a-adb4-48e5-8382-961472075bc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:20.649999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:20.677607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1340/2711] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:20.682131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.743018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9fbe30-5060-4b72-b2bd-245320682ecb admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.748516 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.749175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.749462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.749741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.750432 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.752321 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.776824 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1372/2712] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:20 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:21:20.780878 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683148a-adb4-48e5-8382-961472075bc1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.792070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1683148a-adb4-48e5-8382-961472075bc1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.847948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.848580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.848868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.849132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.849801 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.851701 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.871513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1341/2713] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:20 2022] POST /identity/v3/users => generated 293 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:20.875497 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.886164 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995f991-0e6f-4362-a094-38294db735b2 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.890809 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.891461 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.891747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.892028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.892830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.894441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.906138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1373/2714] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:21:20.910067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.920579 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6549b40d-0ac7-401c-aa25-f5fad1db7aa0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.926072 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5ff579-f992-4de6-b307-9b4cb78ad8f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.927004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5ff579-f992-4de6-b307-9b4cb78ad8f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.927462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5ff579-f992-4de6-b307-9b4cb78ad8f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.927869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5ff579-f992-4de6-b307-9b4cb78ad8f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.928652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5ff579-f992-4de6-b307-9b4cb78ad8f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.930320 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5ff579-f992-4de6-b307-9b4cb78ad8f4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.940687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1342/2715] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:20.944813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:20.951666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:20.955369 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6a055-0dc1-4c3b-9cfd-392336cabb9d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:20.961297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:20.962338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:20.962797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:20.963135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] PATH_INFO: `/v3/projects/2fac30b128224c359a3466e4e1c0502c/users/57b0bcd8b1f04cfabeae23e92aa19244/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:20.979366 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2fac30b128224c359a3466e4e1c0502c, user_id=57b0bcd8b1f04cfabeae23e92aa19244, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:20.982140 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:20.996090 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1374/2716] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:20 2022] PUT /identity/v3/projects/2fac30b128224c359a3466e4e1c0502c/users/57b0bcd8b1f04cfabeae23e92aa19244/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:21.007150 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1bc03-c713-44a8-9bf1-3f4d80b668f0 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.007442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1bc03-c713-44a8-9bf1-3f4d80b668f0 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.007717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da1bc03-c713-44a8-9bf1-3f4d80b668f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.034845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-4da1bc03-c713-44a8-9bf1-3f4d80b668f0 None None] MFA Rules not processed for user `57b0bcd8b1f04cfabeae23e92aa19244`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:21.036057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4da1bc03-c713-44a8-9bf1-3f4d80b668f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:21.099937 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1343/2717] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:21.104216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:21.165692 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d37218-a79f-45aa-8262-8df2a68a9931 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:21.170288 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:21.170936 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.171216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.171491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.172174 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:21.174101 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:21.199206 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1375/2718] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:21 2022] POST /identity/v3/projects => generated 387 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:21.203208 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1bc03-c713-44a8-9bf1-3f4d80b668f0 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:21.213659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da1bc03-c713-44a8-9bf1-3f4d80b668f0 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:21.256183 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:21.270222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:21.270894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.271181 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.271462 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.272156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:21.274158 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:21.292939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1344/2719] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:21 2022] POST /identity/v3/users => generated 300 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:21.296934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:21.307584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805309d4-7fde-4fef-a618-045cefc67625 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:21.312198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:21.312909 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.313194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.313468 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.314213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:21.315840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:21.327048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1376/2720] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:21.331083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:21.340908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e87a93-1ece-4e5b-83a0-1ac04fa1783a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:21.345442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:21.346315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.346600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.346880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] PATH_INFO: `/v3/projects/07a7421b04874347a071b673fa16682f/users/28a014fa9fab472a9336f9d2071d5a3f/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.361982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=07a7421b04874347a071b673fa16682f, user_id=28a014fa9fab472a9336f9d2071d5a3f, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:21.364599 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:21.378125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1345/2721] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:21 2022] PUT /identity/v3/projects/07a7421b04874347a071b673fa16682f/users/28a014fa9fab472a9336f9d2071d5a3f/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:21.382005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:21.392001 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56e7555-2514-476c-8e11-55c73bd733e8 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:21.447712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d7b83c-3c61-4545-b0d5-4c5a64c02cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:21.448382 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d7b83c-3c61-4545-b0d5-4c5a64c02cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.448695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d7b83c-3c61-4545-b0d5-4c5a64c02cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.448975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d7b83c-3c61-4545-b0d5-4c5a64c02cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.449698 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d7b83c-3c61-4545-b0d5-4c5a64c02cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:21.451302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d7b83c-3c61-4545-b0d5-4c5a64c02cb9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:21.461014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1377/2722] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:21 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:21.465195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:21.475542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804d882-8ccb-43b1-9a5e-9c93bf9c4a8f admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:21.480653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:21.481722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.482016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.482291 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] PATH_INFO: `/v3/projects/07a7421b04874347a071b673fa16682f/users/28a014fa9fab472a9336f9d2071d5a3f/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.484237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07a7421b04874347a071b673fa16682f, user_id=28a014fa9fab472a9336f9d2071d5a3f, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:21.486890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:21.497905 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1346/2723] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:21 2022] PUT /identity/v3/projects/07a7421b04874347a071b673fa16682f/users/28a014fa9fab472a9336f9d2071d5a3f/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:21.507966 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.508259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.508609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.534494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] MFA Rules not processed for user `28a014fa9fab472a9336f9d2071d5a3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:21.535704 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:21.575942 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:21.604840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1378/2724] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:21 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:21.615413 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:21.617064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:21.674610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb7ffcb-a22e-4621-a878-75b71e359415 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:21.678281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:21.678919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:21.679205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:21.679491 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:21.688907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:21.690334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:21.700289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1347/2725] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:21 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:22.178072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:22.179506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:22.191569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c8814e-378f-4cad-af1d-78270f27ea16 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:22.195173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:22.195847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:22.196149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:22.196438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:22.256178 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:22.257707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:22.267398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1379/2726] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:26.213265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:26.221830 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eddbf0c-a46f-4b3d-bdb0-ce18a79c537c service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:26.225499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adee4f42-c919-4db4-98fd-73abb9e2949e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:26.226127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee4f42-c919-4db4-98fd-73abb9e2949e service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:26.226409 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee4f42-c919-4db4-98fd-73abb9e2949e service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:26.226684 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adee4f42-c919-4db4-98fd-73abb9e2949e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:26.283453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adee4f42-c919-4db4-98fd-73abb9e2949e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:26.284941 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adee4f42-c919-4db4-98fd-73abb9e2949e service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:26.293828 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1348/2727] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:26 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:28.325501 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:28.387991 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e6c487-07fa-4a52-bafb-d9e14cfbdc28 service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:28.392753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:28.393622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:28.393908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:28.394188 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] PATH_INFO: `/v3/users/e6563259190c47b88baf33c10d76ffde` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:28.395429 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6563259190c47b88baf33c10d76ffde)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:28.397336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:28.441846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6563259190c47b88baf33c10d76ffde'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:28.442507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6563259190c47b88baf33c10d76ffde'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:28.449935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6563259190c47b88baf33c10d76ffde'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:28.450563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6563259190c47b88baf33c10d76ffde'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:28.457027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6563259190c47b88baf33c10d76ffde'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:28.467614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:28.476976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1380/2728] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:28 2022] DELETE /identity/v3/users/e6563259190c47b88baf33c10d76ffde => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:28.481632 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:28.482107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:28.482226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:28.500142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:28.501376 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:28.533470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:28.560250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1349/2729] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:28.570642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:28.572048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:28.630286 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b031589d-26b7-44d1-a716-680fa34c2bc7 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:28.634002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:28.634649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:28.634933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:28.635213 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:28.644104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:28.645761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:28.655650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1381/2730] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:28.925417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:28.929260 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:28.943097 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6917f7-e257-4704-8a1f-5fa90bb3df53 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:28.947844 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:28.948747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:28.949162 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:28.949444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] PATH_INFO: `/v3/projects/619488d546174e1cb3a7bfe566eef450` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:28.950641 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] RBAC: Authorizing `identity:delete_project(project_id=619488d546174e1cb3a7bfe566eef450)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:28.952711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:28.999893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '619488d546174e1cb3a7bfe566eef450', 'request_id': 'req-55c62520-044e-4ee4-80b8-d7b278f3c675'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.005437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '619488d546174e1cb3a7bfe566eef450', 'request_id': 'req-55c62520-044e-4ee4-80b8-d7b278f3c675'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.011567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '619488d546174e1cb3a7bfe566eef450', 'request_id': 'req-55c62520-044e-4ee4-80b8-d7b278f3c675'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.023537 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] The token cache is being invalidate because project 619488d546174e1cb3a7bfe566eef450 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:29.023829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.024566 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.026202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1350/2731] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:28 2022] DELETE /identity/v3/projects/619488d546174e1cb3a7bfe566eef450 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:29.030171 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:29.040518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d401b6-1e88-4d31-b29b-7338061d8ad4 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:29.095667 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:29.096571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:29.096859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:29.097139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] PATH_INFO: `/v3/users/57b0bcd8b1f04cfabeae23e92aa19244` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:29.098317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57b0bcd8b1f04cfabeae23e92aa19244)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:29.100231 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:29.147791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57b0bcd8b1f04cfabeae23e92aa19244'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.148467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57b0bcd8b1f04cfabeae23e92aa19244'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.155247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57b0bcd8b1f04cfabeae23e92aa19244'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.155863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57b0bcd8b1f04cfabeae23e92aa19244'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.163123 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57b0bcd8b1f04cfabeae23e92aa19244'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.180077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1382/2732] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:29 2022] DELETE /identity/v3/users/57b0bcd8b1f04cfabeae23e92aa19244 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:29.443492 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:29.444957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:29.458022 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c62520-044e-4ee4-80b8-d7b278f3c675 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:29.515152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:29.516017 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:29.516302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:29.516627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] PATH_INFO: `/v3/projects/2fac30b128224c359a3466e4e1c0502c` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:29.517833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2fac30b128224c359a3466e4e1c0502c)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:29.519754 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:29.565672 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2fac30b128224c359a3466e4e1c0502c', 'request_id': 'req-8c636726-6ab7-439a-ba38-f8dabe990c93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.571212 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2fac30b128224c359a3466e4e1c0502c', 'request_id': 'req-8c636726-6ab7-439a-ba38-f8dabe990c93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.577335 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2fac30b128224c359a3466e4e1c0502c', 'request_id': 'req-8c636726-6ab7-439a-ba38-f8dabe990c93'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.579816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:29.590568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] The token cache is being invalidate because project 2fac30b128224c359a3466e4e1c0502c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:29.590859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.591475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.593061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1351/2733] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:29 2022] DELETE /identity/v3/projects/2fac30b128224c359a3466e4e1c0502c => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:29.597064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:29.598471 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:29.611512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92dcb7d1-ecc5-4961-98c1-4e9483a6bc63 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:29.622811 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:29.669452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:29.670338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:29.670625 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:29.670904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] PATH_INFO: `/v3/users/28a014fa9fab472a9336f9d2071d5a3f` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:29.672087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] RBAC: Authorizing `identity:delete_user(user_id=28a014fa9fab472a9336f9d2071d5a3f)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:29.674081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:29.719257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28a014fa9fab472a9336f9d2071d5a3f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.719833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28a014fa9fab472a9336f9d2071d5a3f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.726768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28a014fa9fab472a9336f9d2071d5a3f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.727404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28a014fa9fab472a9336f9d2071d5a3f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.734234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-253a8ae9-1347-4c54-b1f9-ca539a11ef9f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28a014fa9fab472a9336f9d2071d5a3f'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:29.751648 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1383/2734] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:29 2022] DELETE /identity/v3/users/28a014fa9fab472a9336f9d2071d5a3f => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:30.018816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.020513 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:30.038744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c636726-6ab7-439a-ba38-f8dabe990c93 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.094362 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.095253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.095536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.095812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] PATH_INFO: `/v3/projects/07a7421b04874347a071b673fa16682f` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.096997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07a7421b04874347a071b673fa16682f)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.098895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.146009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07a7421b04874347a071b673fa16682f', 'request_id': 'req-885f738c-c0d1-4a09-a76b-ef653a607816'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:30.152104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07a7421b04874347a071b673fa16682f', 'request_id': 'req-885f738c-c0d1-4a09-a76b-ef653a607816'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:30.158131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07a7421b04874347a071b673fa16682f', 'request_id': 'req-885f738c-c0d1-4a09-a76b-ef653a607816'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:30.170618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] The token cache is being invalidate because project 07a7421b04874347a071b673fa16682f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:30.170906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:30.171548 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-885f738c-c0d1-4a09-a76b-ef653a607816 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:30.173151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1352/2735] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:30 2022] DELETE /identity/v3/projects/07a7421b04874347a071b673fa16682f => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:30.179738 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c465e-87ef-492d-ab2d-af7f67b0c778 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.180029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c465e-87ef-492d-ab2d-af7f67b0c778 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.180307 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c465e-87ef-492d-ab2d-af7f67b0c778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.198146 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-895c465e-87ef-492d-ab2d-af7f67b0c778 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:30.199350 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-895c465e-87ef-492d-ab2d-af7f67b0c778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:30.257585 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1384/2736] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:30.265896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea11f707-9284-4ed7-907a-ede36ef3eddf None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.266134 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea11f707-9284-4ed7-907a-ede36ef3eddf None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.266415 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea11f707-9284-4ed7-907a-ede36ef3eddf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.284820 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ea11f707-9284-4ed7-907a-ede36ef3eddf None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:30.286028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ea11f707-9284-4ed7-907a-ede36ef3eddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:30.294422 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1353/2737] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:30.298465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895c465e-87ef-492d-ab2d-af7f67b0c778 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.308087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895c465e-87ef-492d-ab2d-af7f67b0c778 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.312671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.313322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.313604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.313880 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.314627 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:21:30.314935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.316557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.327967 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1385/2738] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:21:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:21:30.332031 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea11f707-9284-4ed7-907a-ede36ef3eddf None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.341703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea11f707-9284-4ed7-907a-ede36ef3eddf None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.346349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.346990 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.347304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.347587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.348272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.350806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.376489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1354/2739] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:30 2022] POST /identity/v3/projects => generated 387 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:30.380379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.390589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124b65e4-9610-459f-82e6-cdfbeee62d26 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.409389 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:30.448045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.448733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.449020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.449297 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.449979 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.451891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.471229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1386/2740] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:30 2022] POST /identity/v3/users => generated 294 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:30.475321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.485546 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af18169-8807-4b5a-ae57-222f2209b7a8 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.490427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.491087 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.491374 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.491652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.492375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.494005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.504273 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1355/2741] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:30.508584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.518469 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d835ed-4cef-4b67-b84f-2460f7ff0457 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.523574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459ad7ae-1bfa-47d8-a66b-9d74b1f6aa3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.524219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459ad7ae-1bfa-47d8-a66b-9d74b1f6aa3b admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.524539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459ad7ae-1bfa-47d8-a66b-9d74b1f6aa3b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.524820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459ad7ae-1bfa-47d8-a66b-9d74b1f6aa3b admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.525570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459ad7ae-1bfa-47d8-a66b-9d74b1f6aa3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.527155 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459ad7ae-1bfa-47d8-a66b-9d74b1f6aa3b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.537037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1387/2742] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:30.540839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.551117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048ecab-aebd-4865-8c2a-43ca54d02f50 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.556119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.557019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.557304 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.557580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] PATH_INFO: `/v3/projects/f80c13176069404ea9fc15d12410363d/users/baf322e304a84f959769e885744e6209/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.572666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f80c13176069404ea9fc15d12410363d, user_id=baf322e304a84f959769e885744e6209, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.575323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.588146 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1356/2743] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:30 2022] PUT /identity/v3/projects/f80c13176069404ea9fc15d12410363d/users/baf322e304a84f959769e885744e6209/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:30.600546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ea4e0-0c5b-4693-ba71-c389f3b9080f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.600999 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ea4e0-0c5b-4693-ba71-c389f3b9080f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.601093 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ea4e0-0c5b-4693-ba71-c389f3b9080f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.626830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-492ea4e0-0c5b-4693-ba71-c389f3b9080f None None] MFA Rules not processed for user `baf322e304a84f959769e885744e6209`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:30.628120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-492ea4e0-0c5b-4693-ba71-c389f3b9080f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:30.689977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1388/2744] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:30.695127 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.715973 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:30.758115 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128c9925-a59e-4817-aa83-23b0af079721 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.762744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.763404 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.763696 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.763976 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.764689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.766610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.790843 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1357/2745] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:30 2022] POST /identity/v3/projects => generated 389 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:30.795126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ea4e0-0c5b-4693-ba71-c389f3b9080f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.806638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ea4e0-0c5b-4693-ba71-c389f3b9080f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.863992 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.865003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.865785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.866063 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.866759 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.868732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.889256 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1389/2746] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:30 2022] POST /identity/v3/users => generated 301 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:30.893091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.904276 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fb9ec8-cf3e-417f-bd3f-9f642dff9f0b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.908942 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.909600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.909887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.910168 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.910896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.912545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.924299 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1358/2747] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:30.928868 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.940067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a477a5a9-607d-4950-9dea-8a99cb911b80 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:30.945328 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:30.946209 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:30.946494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:30.946776 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] PATH_INFO: `/v3/projects/efef57cb08f0462d86e33ba81fe16d3f/users/ea86a5cce2504d46b015e27a1815262d/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:30.961482 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efef57cb08f0462d86e33ba81fe16d3f, user_id=ea86a5cce2504d46b015e27a1815262d, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:30.964161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:30.967168 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:30.978181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1390/2748] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:30 2022] PUT /identity/v3/projects/efef57cb08f0462d86e33ba81fe16d3f/users/ea86a5cce2504d46b015e27a1815262d/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:30.982096 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:30.992742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4d8e4-d582-46db-9b74-530be52e8d7c admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:31.047859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06384793-01cc-42c7-8c58-36ad7b8236e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:31.048476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06384793-01cc-42c7-8c58-36ad7b8236e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:31.048758 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06384793-01cc-42c7-8c58-36ad7b8236e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:31.049029 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06384793-01cc-42c7-8c58-36ad7b8236e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:31.049742 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06384793-01cc-42c7-8c58-36ad7b8236e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:31.051332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06384793-01cc-42c7-8c58-36ad7b8236e2 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:31.061255 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1359/2749] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:30 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:31.065386 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:31.076119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a31db2d-2ed3-46ce-b094-43a9b49e1054 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:31.080910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:31.081796 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:31.082077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:31.082352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] PATH_INFO: `/v3/projects/efef57cb08f0462d86e33ba81fe16d3f/users/ea86a5cce2504d46b015e27a1815262d/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:31.084406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efef57cb08f0462d86e33ba81fe16d3f, user_id=ea86a5cce2504d46b015e27a1815262d, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:31.087057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:31.097208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1391/2750] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:31 2022] PUT /identity/v3/projects/efef57cb08f0462d86e33ba81fe16d3f/users/ea86a5cce2504d46b015e27a1815262d/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:31.107659 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05557c50-d031-419d-892c-4b447dd131e7 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:31.107949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05557c50-d031-419d-892c-4b447dd131e7 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:31.108228 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05557c50-d031-419d-892c-4b447dd131e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:31.134231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-05557c50-d031-419d-892c-4b447dd131e7 None None] MFA Rules not processed for user `ea86a5cce2504d46b015e27a1815262d`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:31.135426 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-05557c50-d031-419d-892c-4b447dd131e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:31.202114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1360/2751] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:31 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:31.212842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:31.214581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:31.271571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad1bd6b-176f-4205-9d7d-2fe56ac60635 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:31.275183 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5357b304-6e1a-43cf-baa0-ca47a563bea4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:31.275818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5357b304-6e1a-43cf-baa0-ca47a563bea4 service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:31.276109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5357b304-6e1a-43cf-baa0-ca47a563bea4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:31.276380 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5357b304-6e1a-43cf-baa0-ca47a563bea4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:31.285258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5357b304-6e1a-43cf-baa0-ca47a563bea4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:31.286681 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5357b304-6e1a-43cf-baa0-ca47a563bea4 service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:31.296614 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1392/2752] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:31 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:31.419396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05557c50-d031-419d-892c-4b447dd131e7 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:31.436850 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:31.482364 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05557c50-d031-419d-892c-4b447dd131e7 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:31.486829 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:31.487693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:31.487977 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:31.488251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] PATH_INFO: `/v3/users/baf322e304a84f959769e885744e6209` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:31.489743 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=baf322e304a84f959769e885744e6209)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:31.491755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:31.536165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baf322e304a84f959769e885744e6209'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:31.543543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baf322e304a84f959769e885744e6209'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:31.544194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baf322e304a84f959769e885744e6209'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:31.544827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baf322e304a84f959769e885744e6209'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:31.556545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baf322e304a84f959769e885744e6209'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:31.569610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1361/2753] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:31 2022] DELETE /identity/v3/users/baf322e304a84f959769e885744e6209 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:31.574149 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:31.574464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:31.574748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:31.592091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:31.593303 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:31.650778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1393/2754] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:31.660328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:31.661722 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:31.717040 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0976b7-6243-43f6-b898-8a19294dca35 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:31.720708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:31.721358 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:31.721640 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:31.721915 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:31.732210 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:31.733660 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:31.743216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1362/2755] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:21:32.237173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:32.238735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:32.252294 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b701b6c-75bf-4be6-84b9-2c1e05a1bbca None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:32.257663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:32.258528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:32.258917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:32.259204 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] PATH_INFO: `/v3/projects/f80c13176069404ea9fc15d12410363d` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:32.260513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f80c13176069404ea9fc15d12410363d)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:32.262522 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:32.309191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f80c13176069404ea9fc15d12410363d', 'request_id': 'req-0db57982-279d-4301-8163-55e6cd95a4f3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.314729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f80c13176069404ea9fc15d12410363d', 'request_id': 'req-0db57982-279d-4301-8163-55e6cd95a4f3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.322407 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f80c13176069404ea9fc15d12410363d', 'request_id': 'req-0db57982-279d-4301-8163-55e6cd95a4f3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.333324 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] The token cache is being invalidate because project f80c13176069404ea9fc15d12410363d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:32.333641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.334259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.335820 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1394/2756] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:32 2022] DELETE /identity/v3/projects/f80c13176069404ea9fc15d12410363d => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:32.339955 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:32.349606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8ca535-969f-4cbb-ad46-24d235fb3b64 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:32.382652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:32.406694 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:32.407563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:32.407847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:32.408128 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] PATH_INFO: `/v3/users/ea86a5cce2504d46b015e27a1815262d` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:32.409370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea86a5cce2504d46b015e27a1815262d)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:32.411340 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:32.456121 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea86a5cce2504d46b015e27a1815262d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.462919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea86a5cce2504d46b015e27a1815262d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.463568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea86a5cce2504d46b015e27a1815262d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.464175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea86a5cce2504d46b015e27a1815262d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.475774 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-26cf8408-798c-47ec-b497-dd07ec0d9cde admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea86a5cce2504d46b015e27a1815262d'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.488333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1363/2757] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:21:32 2022] DELETE /identity/v3/users/ea86a5cce2504d46b015e27a1815262d => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:32.753193 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:32.754829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:32.767673 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db57982-279d-4301-8163-55e6cd95a4f3 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:32.772990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:32.827434 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:32.828295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:32.828610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:32.828890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] PATH_INFO: `/v3/projects/efef57cb08f0462d86e33ba81fe16d3f` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:32.830069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=efef57cb08f0462d86e33ba81fe16d3f)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:32.831990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:32.879428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efef57cb08f0462d86e33ba81fe16d3f', 'request_id': 'req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.884931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efef57cb08f0462d86e33ba81fe16d3f', 'request_id': 'req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.892071 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efef57cb08f0462d86e33ba81fe16d3f', 'request_id': 'req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.902976 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] The token cache is being invalidate because project efef57cb08f0462d86e33ba81fe16d3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:21:32.903265 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.903899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-a4a8f2fb-d216-4529-b3ca-f5271e2d0d8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:21:32.905459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1395/2758] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:21:32 2022] DELETE /identity/v3/projects/efef57cb08f0462d86e33ba81fe16d3f => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:32.910636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f278f-6989-413b-8dfb-3eb508e5dc16 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:32.911116 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f278f-6989-413b-8dfb-3eb508e5dc16 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:32.911747 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f278f-6989-413b-8dfb-3eb508e5dc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:32.930685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-395f278f-6989-413b-8dfb-3eb508e5dc16 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:32.931900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-395f278f-6989-413b-8dfb-3eb508e5dc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:32.990943 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1364/2759] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:32.998994 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe9d905-ad5e-4ee7-8b2d-a004472f580e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:32.999224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe9d905-ad5e-4ee7-8b2d-a004472f580e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:32.999500 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe9d905-ad5e-4ee7-8b2d-a004472f580e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.017438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-cfe9d905-ad5e-4ee7-8b2d-a004472f580e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:33.018651 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-cfe9d905-ad5e-4ee7-8b2d-a004472f580e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:33.026975 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1396/2760] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:33.030962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f278f-6989-413b-8dfb-3eb508e5dc16 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:33.041271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f278f-6989-413b-8dfb-3eb508e5dc16 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:33.051218 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:33.051908 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.052195 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.052511 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.053248 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:21:33.053560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:33.055169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:33.067378 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1365/2761] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:21:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:21:33.071711 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe9d905-ad5e-4ee7-8b2d-a004472f580e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:33.080997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe9d905-ad5e-4ee7-8b2d-a004472f580e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:33.086091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:33.086729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.087011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.087284 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.087957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:33.089891 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:33.114850 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1397/2762] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:21:33 2022] POST /identity/v3/projects => generated 367 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:33.119161 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:33.129236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c2c227-1173-4155-b3b9-f16cfdd96545 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:33.172332 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:33.188685 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:33.189354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.189639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.189919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.190608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:33.192562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:33.211810 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1366/2763] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:21:33 2022] POST /identity/v3/users => generated 284 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:21:33.215922 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:33.226761 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b75d57b-6f11-446d-b500-bc0293534517 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:33.231322 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:33.231978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.232249 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.232618 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.233408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:33.235080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:33.245592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1398/2764] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:21:33.249459 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:33.259230 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3f0903-ac03-4827-80f4-be53cac84ea4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:33.263805 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a51b76-bd1c-451e-b179-c47fdaf86d1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:33.264493 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a51b76-bd1c-451e-b179-c47fdaf86d1c admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.264786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a51b76-bd1c-451e-b179-c47fdaf86d1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.265070 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a51b76-bd1c-451e-b179-c47fdaf86d1c admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.265794 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a51b76-bd1c-451e-b179-c47fdaf86d1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:33.267370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a51b76-bd1c-451e-b179-c47fdaf86d1c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:33.277650 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1367/2765] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:21:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 05 10:21:33.281464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:33.292027 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e243f1c1-7690-428c-9472-ce76115b0950 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:33.296875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:33.297749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.298037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.298317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] PATH_INFO: `/v3/projects/6e256c310b6544339ad69ef6f3aab2e9/users/247de8142da34624a51c461105d9900b/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.312623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e256c310b6544339ad69ef6f3aab2e9, user_id=247de8142da34624a51c461105d9900b, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:33.315270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:33.328557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1399/2766] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:21:33 2022] PUT /identity/v3/projects/6e256c310b6544339ad69ef6f3aab2e9/users/247de8142da34624a51c461105d9900b/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:21:33.392874 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4d82f-b9a4-445a-8867-295a4a1455bd None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.393169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4d82f-b9a4-445a-8867-295a4a1455bd None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.393448 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4d82f-b9a4-445a-8867-295a4a1455bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.419921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-96a4d82f-b9a4-445a-8867-295a4a1455bd None None] MFA Rules not processed for user `247de8142da34624a51c461105d9900b`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:21:33.421229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-96a4d82f-b9a4-445a-8867-295a4a1455bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:33.489997 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1368/2767] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:21:33 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:21:33.501710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:21:33.503323 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:21:33.544072 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:33.562695 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6551787-ef38-4647-83ef-80460434b1a0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:21:33.566335 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b0dede-336c-417d-8f81-c21a50e04372 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:21:33.566985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0dede-336c-417d-8f81-c21a50e04372 service glance] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:21:33.567807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0dede-336c-417d-8f81-c21a50e04372 service glance] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:21:33.568087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b0dede-336c-417d-8f81-c21a50e04372 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:21:33.576281 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b0dede-336c-417d-8f81-c21a50e04372 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:21:33.577747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b0dede-336c-417d-8f81-c21a50e04372 service glance] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:21:33.587091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1400/2768] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:21:33 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:21:43.486812 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=46,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:21:43.582444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=10 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:18.248222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4d82f-b9a4-445a-8867-295a4a1455bd None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:18.323181 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4d82f-b9a4-445a-8867-295a4a1455bd None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:18.327713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:18.328628 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:18.328924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:18.329201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] PATH_INFO: `/v3/users/247de8142da34624a51c461105d9900b` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:18.330400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=247de8142da34624a51c461105d9900b)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:18.332328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:18.383045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '247de8142da34624a51c461105d9900b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.390282 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '247de8142da34624a51c461105d9900b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.390965 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '247de8142da34624a51c461105d9900b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.391563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '247de8142da34624a51c461105d9900b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.404622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '247de8142da34624a51c461105d9900b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.419698 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1369/2769] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:22:18 2022] DELETE /identity/v3/users/247de8142da34624a51c461105d9900b => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:22:18.424060 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:18.424603 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:18.424792 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:18.443220 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:18.444541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:18.506749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1401/2770] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:22:18.516993 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:18.518587 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:18.581761 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082706cc-a537-4393-9243-696514dad4ba admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:18.586020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086996e-1dfa-41ab-a9a8-8511f5b882e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:18.586681 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086996e-1dfa-41ab-a9a8-8511f5b882e8 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:18.586969 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086996e-1dfa-41ab-a9a8-8511f5b882e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:18.587251 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086996e-1dfa-41ab-a9a8-8511f5b882e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:18.596988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086996e-1dfa-41ab-a9a8-8511f5b882e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:18.599253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086996e-1dfa-41ab-a9a8-8511f5b882e8 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:18.613412 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1370/2771] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:22:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:22:18.881646 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:18.883111 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:18.895823 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab134537-2a98-4b44-97ec-bc4ccb41f01e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:18.900548 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:18.901422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:18.901707 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:18.901985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] PATH_INFO: `/v3/projects/6e256c310b6544339ad69ef6f3aab2e9` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:18.903218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e256c310b6544339ad69ef6f3aab2e9)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:18.905184 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:18.965566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e256c310b6544339ad69ef6f3aab2e9', 'request_id': 'req-5d076b4c-cd73-42ce-b682-ac81c44f143b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.971677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e256c310b6544339ad69ef6f3aab2e9', 'request_id': 'req-5d076b4c-cd73-42ce-b682-ac81c44f143b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.981539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e256c310b6544339ad69ef6f3aab2e9', 'request_id': 'req-5d076b4c-cd73-42ce-b682-ac81c44f143b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.994629 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] The token cache is being invalidate because project 6e256c310b6544339ad69ef6f3aab2e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:22:18.994980 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.995588 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-5d076b4c-cd73-42ce-b682-ac81c44f143b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:18.997960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1402/2772] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:22:18 2022] DELETE /identity/v3/projects/6e256c310b6544339ad69ef6f3aab2e9 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:22:19.002913 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2eab69-91f4-432f-bdc0-e01ffe45a900 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.003215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2eab69-91f4-432f-bdc0-e01ffe45a900 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.003495 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2eab69-91f4-432f-bdc0-e01ffe45a900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.024238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fb2eab69-91f4-432f-bdc0-e01ffe45a900 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:19.025499 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fb2eab69-91f4-432f-bdc0-e01ffe45a900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:19.043948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:19.089603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1371/2773] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:19.097763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd6877-7dcd-499c-9de8-3156f6a92362 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.098076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd6877-7dcd-499c-9de8-3156f6a92362 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.098371 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fd6877-7dcd-499c-9de8-3156f6a92362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.119148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-92fd6877-7dcd-499c-9de8-3156f6a92362 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:19.120459 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-92fd6877-7dcd-499c-9de8-3156f6a92362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:19.128662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1403/2774] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:19.133420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2eab69-91f4-432f-bdc0-e01ffe45a900 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:19.143505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2eab69-91f4-432f-bdc0-e01ffe45a900 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:19.148676 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:19.149348 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.149652 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.149922 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.150653 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:22:19.150963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:19.152597 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:19.165030 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1372/2775] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:22:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:22:19.169638 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd6877-7dcd-499c-9de8-3156f6a92362 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:19.179971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fd6877-7dcd-499c-9de8-3156f6a92362 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:19.184571 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:19.185224 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.185505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.185810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.186486 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:19.188370 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:19.215136 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1404/2776] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:22:19 2022] POST /identity/v3/projects => generated 399 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:22:19.219281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:19.230476 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614c8f4-a8de-4281-9d5f-49637ad785b3 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:19.290354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:19.292618 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.293126 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.293559 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.294600 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:19.297689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:19.321703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1373/2777] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:22:19 2022] POST /identity/v3/users => generated 300 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:22:19.326428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:19.338308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb7878-daab-418c-a638-ce750013e175 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:19.344581 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdc6ed5-cba3-4c5b-896b-0850f5986600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:19.344847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdc6ed5-cba3-4c5b-896b-0850f5986600 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.345314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdc6ed5-cba3-4c5b-896b-0850f5986600 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.345452 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdc6ed5-cba3-4c5b-896b-0850f5986600 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.346218 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdc6ed5-cba3-4c5b-896b-0850f5986600 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:19.348192 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdc6ed5-cba3-4c5b-896b-0850f5986600 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:19.361113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1405/2778] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:22:19.365854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:19.378818 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9857ac-4520-4d22-9ac4-b61ea4d7e1d4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:19.381235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.381568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.381898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.382998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:19.384005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.384356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.384606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.385352 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:19.387334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:19.397893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1374/2779] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:22:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:22:19.401125 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] MFA Rules not processed for user `d4a608563189457dbcadf8d48acdd84a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:19.401873 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:19.402368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:19.412106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04177313-9557-4859-bdec-fce7043be2d0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:19.416834 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66a37c8-d4af-41e4-b2a7-ffe46412148c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:19.417759 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66a37c8-d4af-41e4-b2a7-ffe46412148c admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.418173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66a37c8-d4af-41e4-b2a7-ffe46412148c admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.418327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66a37c8-d4af-41e4-b2a7-ffe46412148c admin admin] PATH_INFO: `/v3/projects/d3f7553151934839bee45038b35a46b6/users/677edc27ba8d496f9ce26b82774c4952/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.434265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66a37c8-d4af-41e4-b2a7-ffe46412148c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3f7553151934839bee45038b35a46b6, user_id=677edc27ba8d496f9ce26b82774c4952, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:19.436954 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66a37c8-d4af-41e4-b2a7-ffe46412148c admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:19.449458 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:19.450665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1375/2780] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:22:19 2022] PUT /identity/v3/projects/d3f7553151934839bee45038b35a46b6/users/677edc27ba8d496f9ce26b82774c4952/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:22:19.462718 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.463008 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.463292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.466720 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1406/2781] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:19.493657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] MFA Rules not processed for user `677edc27ba8d496f9ce26b82774c4952`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:19.494869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:19.552522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:19.563960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1376/2782] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:19 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:22:19.575457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:19.576842 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:19.637069 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46914b93-522b-4cbf-a722-c3fff59c5d6f None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:19.687887 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:19.688765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:19.688876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:19.689105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:19.697873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:19.699474 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:19.708962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1407/2783] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:22:19 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:22:20.102370 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:20.104010 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:20.188246 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84c5c11-f557-487f-8c48-bd982d4af1e9 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:20.193779 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d39026f-5884-41cc-8713-df5ce4118286 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:20.194436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39026f-5884-41cc-8713-df5ce4118286 service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:20.194730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39026f-5884-41cc-8713-df5ce4118286 service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:20.195023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d39026f-5884-41cc-8713-df5ce4118286 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:20.196875 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8d39026f-5884-41cc-8713-df5ce4118286 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:20.209603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39026f-5884-41cc-8713-df5ce4118286 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:20.211262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d39026f-5884-41cc-8713-df5ce4118286 service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:20.221215 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1377/2784] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:22:20 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:22:27.843777 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:27.905352 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f623e6-4afb-493b-b4cc-a94a96cac03d service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:27.910408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:27.911268 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:27.911553 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:27.911833 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] PATH_INFO: `/v3/users/677edc27ba8d496f9ce26b82774c4952` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:27.913028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=677edc27ba8d496f9ce26b82774c4952)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:27.914948 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:27.962696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '677edc27ba8d496f9ce26b82774c4952'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:27.963368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '677edc27ba8d496f9ce26b82774c4952'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:27.965464 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:27.972703 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '677edc27ba8d496f9ce26b82774c4952'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:27.973438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '677edc27ba8d496f9ce26b82774c4952'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:27.981223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '677edc27ba8d496f9ce26b82774c4952'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:27.999641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1408/2785] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:22:27 2022] DELETE /identity/v3/users/677edc27ba8d496f9ce26b82774c4952 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:22:28.004100 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.004675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.004675 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.022487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:28.023703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:28.086312 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1378/2786] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:28.096848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.098270 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:28.156559 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e50a26f-d0f3-4de6-8781-4bfce5f0cea2 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.160920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc1f97c-6288-4912-b9cd-7e4516345df9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.161574 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc1f97c-6288-4912-b9cd-7e4516345df9 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.161859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc1f97c-6288-4912-b9cd-7e4516345df9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.162135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc1f97c-6288-4912-b9cd-7e4516345df9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.171440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc1f97c-6288-4912-b9cd-7e4516345df9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.172911 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc1f97c-6288-4912-b9cd-7e4516345df9 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:28.182356 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1409/2787] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:22:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:22:28.461084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.462502 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:28.480661 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25430477-ffc1-4103-9232-6e959a732dd1 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.489391 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.490524 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.490916 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.491452 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] PATH_INFO: `/v3/projects/d3f7553151934839bee45038b35a46b6` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.493192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3f7553151934839bee45038b35a46b6)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.495549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:28.544638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3f7553151934839bee45038b35a46b6', 'request_id': 'req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:28.550804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3f7553151934839bee45038b35a46b6', 'request_id': 'req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:28.557815 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3f7553151934839bee45038b35a46b6', 'request_id': 'req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:28.570656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] The token cache is being invalidate because project d3f7553151934839bee45038b35a46b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:22:28.570948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:28.572184 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-7ba0f0ca-b175-463c-9158-69d1d8dce6dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:22:28.573823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1379/2788] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:22:28 2022] DELETE /identity/v3/projects/d3f7553151934839bee45038b35a46b6 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:22:28.580908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea57153-219c-4fcb-870c-ef7b38303e9c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.581168 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea57153-219c-4fcb-870c-ef7b38303e9c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.581445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea57153-219c-4fcb-870c-ef7b38303e9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.600005 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4ea57153-219c-4fcb-870c-ef7b38303e9c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:28.601259 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4ea57153-219c-4fcb-870c-ef7b38303e9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:28.662046 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1410/2789] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:28.669925 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f264887-10d6-4dec-9225-02de89267e3e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.670221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f264887-10d6-4dec-9225-02de89267e3e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.670505 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f264887-10d6-4dec-9225-02de89267e3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.683443 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:28.689663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-5f264887-10d6-4dec-9225-02de89267e3e None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:28.691770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-5f264887-10d6-4dec-9225-02de89267e3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:28.700319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1380/2790] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:28.705108 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea57153-219c-4fcb-870c-ef7b38303e9c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.714884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea57153-219c-4fcb-870c-ef7b38303e9c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.719839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.720525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.720818 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.721098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.721828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:22:28.722138 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.723742 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:28.735163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1411/2791] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:22:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:22:28.739217 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f264887-10d6-4dec-9225-02de89267e3e None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.748333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f264887-10d6-4dec-9225-02de89267e3e None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.753442 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.754081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.754361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.754634 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.755308 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.757229 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:28.782986 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1381/2792] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:22:28 2022] POST /identity/v3/projects => generated 377 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:22:28.787225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.797373 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447810a1-f23b-47a7-8b71-c46bbec91af3 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.848238 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:28.854888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.856000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.856408 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.856724 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.857798 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.859954 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:28.880904 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1412/2793] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:22:28 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:22:28.885049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.895394 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d3594-84e2-43c1-ba8b-913c709e494a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.900103 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.900804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.901091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.901373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.902098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.903706 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:28.914046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1382/2794] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:22:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:22:28.917758 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.931802 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1e0738-609e-45a7-97c6-c34df2304998 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.936513 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a6b2d8-8439-4f3c-972c-fa3942a6456c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.937176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6b2d8-8439-4f3c-972c-fa3942a6456c admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.937462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6b2d8-8439-4f3c-972c-fa3942a6456c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.937741 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a6b2d8-8439-4f3c-972c-fa3942a6456c admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.939097 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a6b2d8-8439-4f3c-972c-fa3942a6456c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.940733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a6b2d8-8439-4f3c-972c-fa3942a6456c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:28.950643 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1413/2795] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:22:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:22:28.954778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:28.964869 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f0914c-880f-4c7d-9b49-be8352e73ff0 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:28.969823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:28.970708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:28.970994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:28.971275 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] PATH_INFO: `/v3/projects/86eac45249ad4a5e9513fcc091bcf42e/users/c9203890ad7649f897b13ac7bdf6be3b/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:28.987416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] RBAC: Authorizing `identity:create_grant(project_id=86eac45249ad4a5e9513fcc091bcf42e, user_id=c9203890ad7649f897b13ac7bdf6be3b, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:28.990131 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:29.003112 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1383/2796] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:22:28 2022] PUT /identity/v3/projects/86eac45249ad4a5e9513fcc091bcf42e/users/c9203890ad7649f897b13ac7bdf6be3b/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:22:29.008148 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6f1dc-8d70-4319-b5ac-37c737372778 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:29.008491 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6f1dc-8d70-4319-b5ac-37c737372778 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:29.008779 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b6f1dc-8d70-4319-b5ac-37c737372778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:29.025983 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-15b6f1dc-8d70-4319-b5ac-37c737372778 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:29.027176 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-15b6f1dc-8d70-4319-b5ac-37c737372778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:29.084313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1414/2797] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:29.093872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:29.095924 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:29.152799 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab1bf8-d463-4d03-aca9-08c1143b767a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:29.156444 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:29.157630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:29.157923 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:29.158204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:29.166974 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:29.168410 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:29.178223 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1384/2798] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:22:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:22:33.111978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95f935-7987-4173-829e-b30f9b8101d9 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:33.112385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95f935-7987-4173-829e-b30f9b8101d9 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:33.112797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d95f935-7987-4173-829e-b30f9b8101d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:33.161939 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-9d95f935-7987-4173-829e-b30f9b8101d9 None None] MFA Rules not processed for user `c9203890ad7649f897b13ac7bdf6be3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:33.163363 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-9d95f935-7987-4173-829e-b30f9b8101d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:33.296869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1415/2799] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:33 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:33.297703 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:33.311205 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c591bdc5-2aeb-4b9d-88ff-9b5955960515 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:33.317114 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:33.318408 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:33.318912 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:33.319192 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:33.319889 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:33.322824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:33.361344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1385/2800] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:22:33 2022] POST /identity/v3/projects => generated 377 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:22:33.364971 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d95f935-7987-4173-829e-b30f9b8101d9 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:33.378346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d95f935-7987-4173-829e-b30f9b8101d9 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:33.443172 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:33.443848 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:33.444134 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:33.444438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:33.445151 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:33.447100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:33.458636 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:33.471568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1416/2801] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:22:33 2022] POST /identity/v3/users => generated 295 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:22:33.475303 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:33.495607 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6367d5af-1b92-4c25-9120-f663895a1922 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:33.500939 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:33.501859 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:33.502377 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:33.502753 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:33.503557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:33.505455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:33.520557 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1386/2802] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:22:33 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:22:33.524765 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:33.535602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee522b61-fdf8-48cf-a7b4-6deb21ef179c admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:33.540613 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:33.541534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:33.541821 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:33.542105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] PATH_INFO: `/v3/projects/78c9221526824231babe7a480dae7bef/users/888de2e07aca4761bed43fb8b8e5f38a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:33.557604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78c9221526824231babe7a480dae7bef, user_id=888de2e07aca4761bed43fb8b8e5f38a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:33.560289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:33.573498 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1417/2803] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:22:33 2022] PUT /identity/v3/projects/78c9221526824231babe7a480dae7bef/users/888de2e07aca4761bed43fb8b8e5f38a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:22:33.578144 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:33.589807 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b093e3-73d4-43c0-af44-db144aaf3847 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:33.648824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d357653-e3f0-42f0-98cf-cea09f2696c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:33.649501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d357653-e3f0-42f0-98cf-cea09f2696c3 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:33.649879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d357653-e3f0-42f0-98cf-cea09f2696c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:33.650165 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d357653-e3f0-42f0-98cf-cea09f2696c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:33.650896 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d357653-e3f0-42f0-98cf-cea09f2696c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:33.652528 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d357653-e3f0-42f0-98cf-cea09f2696c3 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:33.662803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1387/2804] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:22:33 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:22:33.666882 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:33.677120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290b4409-d87d-4fe8-b9cc-19eb3028ad80 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:33.681663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:33.682537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:33.682822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:33.683100 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] PATH_INFO: `/v3/projects/78c9221526824231babe7a480dae7bef/users/888de2e07aca4761bed43fb8b8e5f38a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:33.685076 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] RBAC: Authorizing `identity:create_grant(project_id=78c9221526824231babe7a480dae7bef, user_id=888de2e07aca4761bed43fb8b8e5f38a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:33.688275 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:33.698895 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1418/2805] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:22:33 2022] PUT /identity/v3/projects/78c9221526824231babe7a480dae7bef/users/888de2e07aca4761bed43fb8b8e5f38a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:22:39.511204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169f1ba-ee42-4dbd-8652-632953b1749e None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:39.511689 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169f1ba-ee42-4dbd-8652-632953b1749e None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:39.511789 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169f1ba-ee42-4dbd-8652-632953b1749e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:39.532101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:39.545646 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b169f1ba-ee42-4dbd-8652-632953b1749e None None] MFA Rules not processed for user `888de2e07aca4761bed43fb8b8e5f38a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:22:39.547680 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b169f1ba-ee42-4dbd-8652-632953b1749e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:39.628662 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1388/2806] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:22:39 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:22:39.643876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:22:39.646410 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:22:39.750338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432672b9-2044-46f6-bcca-b65af661d75f admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:22:39.759024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:22:39.759851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:22:39.760302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:22:39.760717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:22:39.822947 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:22:39.824584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:22:39.833977 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1419/2807] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:22:39 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:22:49.625382 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=30 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:22:49.829659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=73,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:23:03.311619 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:23:03.312098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:23:03.312204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:23:03.339712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] MFA Rules not processed for user `c9203890ad7649f897b13ac7bdf6be3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:23:03.340956 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:23:03.363983 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1389/2808] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:23:03 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:23:03.673314 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:23:03.753533 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d36fa-943f-4074-bd83-4511d8f5e6ec service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:23:03.757110 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:23:03.757749 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:23:03.758030 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:23:03.758305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:23:03.760036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:23:03.781591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:23:03.782987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:23:03.791528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1420/2809] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:23:03 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:23:13.362617 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=15 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:23:13.787898 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=44 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:23:16.163920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:23:16.165455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:23:16.188731 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fe131e-35a7-48d0-a779-e48b68473a06 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:23:16.194717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:23:16.195359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:23:16.195644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:23:16.195920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:23:16.197673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:23:16.210395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:23:16.211963 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:23:16.222474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1390/2810] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:23:16 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:23:19.646454 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:23:19.655181 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3c202f-7640-4503-bd32-d767e9305407 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:23:19.658773 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adea6c8-c2ab-4152-bbe7-f435d5517586 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:23:19.659406 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adea6c8-c2ab-4152-bbe7-f435d5517586 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:23:19.659687 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adea6c8-c2ab-4152-bbe7-f435d5517586 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:23:19.659961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adea6c8-c2ab-4152-bbe7-f435d5517586 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:23:19.668816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adea6c8-c2ab-4152-bbe7-f435d5517586 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:23:19.670232 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adea6c8-c2ab-4152-bbe7-f435d5517586 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:23:19.680024 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1421/2811] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:23:19 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:23:26.218067 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=9 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:23:29.675896 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=6 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:23:52.223152 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:23:52.242117 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6fc3e5-73bd-4073-9844-cb38db7ffdd4 service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:23:52.247809 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:23:52.248490 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:23:52.248775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:23:52.249052 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:23:52.250816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:23:52.307235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:23:52.308868 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:23:52.317663 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1391/2812] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:23:52 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:24:02.313591 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=31 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:24:11.974615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:11.975119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:11.975240 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:12.032173 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:24:12.033379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:12.098004 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1422/2813] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:24:11 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:24:12.143295 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:24:12.144775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:12.236666 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b05b578-8a81-4fe0-a6bf-9692308cf35a service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:24:12.241717 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc95fd7e-e88f-48fd-9e02-c413d830d56b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:24:12.242910 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc95fd7e-e88f-48fd-9e02-c413d830d56b service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:12.243193 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc95fd7e-e88f-48fd-9e02-c413d830d56b service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:12.244222 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc95fd7e-e88f-48fd-9e02-c413d830d56b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:12.252760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc95fd7e-e88f-48fd-9e02-c413d830d56b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:24:12.254361 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc95fd7e-e88f-48fd-9e02-c413d830d56b service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:24:12.263106 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1392/2814] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:24:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 05 10:24:22.095215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=44 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:24:22.259854 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=41 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:24:22.587829 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:24:22.589309 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:22.682870 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1909d2-685c-404b-a301-c102affbd50d None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:24:22.688012 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ac6ef9-f229-45b7-a3c0-19ab87b5ba4b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:24:22.688710 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ac6ef9-f229-45b7-a3c0-19ab87b5ba4b service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:22.688997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ac6ef9-f229-45b7-a3c0-19ab87b5ba4b service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:22.689274 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ac6ef9-f229-45b7-a3c0-19ab87b5ba4b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:22.691048 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-99ac6ef9-f229-45b7-a3c0-19ab87b5ba4b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:22.703863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ac6ef9-f229-45b7-a3c0-19ab87b5ba4b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:24:22.705489 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ac6ef9-f229-45b7-a3c0-19ab87b5ba4b service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:24:22.715753 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1423/2815] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:24:22 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:24:26.308019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec67bdc6-be1e-41b7-a85d-5348f25c9bed None None] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:26.308547 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec67bdc6-be1e-41b7-a85d-5348f25c9bed None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:26.308887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec67bdc6-be1e-41b7-a85d-5348f25c9bed None None] PATH_INFO: `/` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:26.310371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1393/2816] 10.222.0.20 () {54 vars in 921 bytes} [Tue Jul 5 10:24:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 05 10:24:26.315647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:26.316353 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:26.316495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:26.344398 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] MFA Rules not processed for user `f99a9fe4b40646d389c3ebf47358956c`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:24:26.345672 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:26.355677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1424/2817] 10.222.0.20 () {58 vars in 1018 bytes} [Tue Jul 5 10:24:26 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:24:26.367893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec67bdc6-be1e-41b7-a85d-5348f25c9bed None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:24:26.369677 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ec67bdc6-be1e-41b7-a85d-5348f25c9bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:26.452991 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec67bdc6-be1e-41b7-a85d-5348f25c9bed None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:24:26.456552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '90d9b9421d454465b8f99137cd182abf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:24:26.457213 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:26.458041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:26.458326 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:26.468519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:24:26.469949 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:24:26.481455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1394/2818] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:24:26 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:24:26.546021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:24:26.547659 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:26.559088 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af6136-6eda-4289-b4e0-33350cefa499 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:24:26.562666 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fdd118-bba0-4d36-9383-8bec268e2a06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:24:26.563302 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fdd118-bba0-4d36-9383-8bec268e2a06 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:26.563589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fdd118-bba0-4d36-9383-8bec268e2a06 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:26.563867 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fdd118-bba0-4d36-9383-8bec268e2a06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:26.565592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-59fdd118-bba0-4d36-9383-8bec268e2a06 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:26.575077 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fdd118-bba0-4d36-9383-8bec268e2a06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:24:26.576546 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fdd118-bba0-4d36-9383-8bec268e2a06 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:24:26.584840 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1425/2819] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:24:26 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:24:26.660227 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:24:26.670400 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23bdc91-c860-45f8-959e-5ead197dc880 service glance] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:24:26.674005 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:24:26.674642 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:26.674927 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:26.675204 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:26.676957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:26.689920 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:24:26.691611 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:24:26.702440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1395/2820] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:24:26 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:24:28.304903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015554e1-8fbf-432c-bede-7ce085ad4341 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:28.305430 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015554e1-8fbf-432c-bede-7ce085ad4341 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:28.305539 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015554e1-8fbf-432c-bede-7ce085ad4341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:28.323602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-015554e1-8fbf-432c-bede-7ce085ad4341 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:24:28.324869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-015554e1-8fbf-432c-bede-7ce085ad4341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:28.331828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1426/2821] 10.222.0.20 () {58 vars in 1009 bytes} [Tue Jul 5 10:24:28 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:24:28.343921 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:24:28.355083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7216e2c-e838-494b-bf44-1ca290d1b140 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:24:28.360064 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:24:28.360744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:28.361033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:28.361319 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:28.369403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:24:28.371018 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:24:28.379734 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1396/2822] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:24:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 05 10:24:38.318551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=57 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:24:38.375900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=47 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:24:46.726875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa8392b-3912-42c2-9bff-dfd2d78898e8 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:46.727336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa8392b-3912-42c2-9bff-dfd2d78898e8 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:46.727475 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa8392b-3912-42c2-9bff-dfd2d78898e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:46.745036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-caa8392b-3912-42c2-9bff-dfd2d78898e8 None None] MFA Rules not processed for user `154e830e318245158269c03fd2d9621a`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:24:46.746254 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-caa8392b-3912-42c2-9bff-dfd2d78898e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:24:46.753402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1427/2823] 10.222.0.20 () {58 vars in 967 bytes} [Tue Jul 5 10:24:46 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:24:46.766533 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:24:46.777009 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1239da6-799d-408c-8519-207e4b5586c4 service swift] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:24:46.782687 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c1d16fdc498e4d84bf8657c7f55d6b26', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:24:46.783317 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:24:46.783603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:24:46.783880 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:24:46.791904 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:24:46.793503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:24:46.801494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1397/2824] 10.222.0.20 () {60 vars in 1384 bytes} [Tue Jul 5 10:24:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 05 10:24:56.739995 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:24:56.798216 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=6 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:25:35.400607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa8392b-3912-42c2-9bff-dfd2d78898e8 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:35.410081 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa8392b-3912-42c2-9bff-dfd2d78898e8 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:35.413597 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbaebe7-81f2-4ad9-8b49-5c6f3153943f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:35.414226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbaebe7-81f2-4ad9-8b49-5c6f3153943f service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:35.414508 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbaebe7-81f2-4ad9-8b49-5c6f3153943f service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:35.414785 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbaebe7-81f2-4ad9-8b49-5c6f3153943f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:35.417145 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-bbbaebe7-81f2-4ad9-8b49-5c6f3153943f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:35.442507 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbaebe7-81f2-4ad9-8b49-5c6f3153943f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:35.443925 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbaebe7-81f2-4ad9-8b49-5c6f3153943f service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:35.464293 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1428/2825] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:35 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:36.426630 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:36.436919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa637ede-fde8-45b6-acaa-8a45c5d73328 service swift] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:36.440501 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:36.441143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:36.441430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:36.441712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:36.443467 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:36.457373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:36.458796 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:36.469732 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1398/2826] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:41.754505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3135db79-53e6-4a34-93c6-9b7ae873ddbb None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:41.755021 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3135db79-53e6-4a34-93c6-9b7ae873ddbb None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:41.755133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3135db79-53e6-4a34-93c6-9b7ae873ddbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:41.785570 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-3135db79-53e6-4a34-93c6-9b7ae873ddbb None None] MFA Rules not processed for user `c9203890ad7649f897b13ac7bdf6be3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:25:41.786883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-3135db79-53e6-4a34-93c6-9b7ae873ddbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:41.794102 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1429/2827] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:25:41 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:25:42.111581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:42.121194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7dca92-92f0-4128-9469-12b8ad3325d1 service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:42.124825 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:42.125471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:42.125755 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:42.126033 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:42.127833 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:42.138151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:42.139571 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:42.151584 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1399/2828] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:42 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:44.322594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3135db79-53e6-4a34-93c6-9b7ae873ddbb None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:44.333223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3135db79-53e6-4a34-93c6-9b7ae873ddbb None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:44.337827 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:44.338495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:44.338780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:44.339057 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:44.340935 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:44.352165 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:44.353668 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:44.363109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1430/2829] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:44 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:44.641804 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:44.651302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a7f58-d084-43da-a40e-4ea2b6170fdd service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:44.654919 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:44.655562 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:44.656342 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:44.656665 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:44.658447 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:44.746847 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:44.748359 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:44.769928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1400/2830] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:48.498615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:48.500098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:48.518317 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35178a5a-4faf-4de8-9586-405499e88f53 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:48.523949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:48.524622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:48.525084 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:48.525361 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:48.527120 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:48.539873 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:48.541609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:48.557641 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1431/2831] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:48 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:48.560521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:48.561931 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:48.574315 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b424683-717d-4cda-8d78-7936b3ee51ca service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:48.580325 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:48.581914 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:48.582201 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:48.582580 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] PATH_INFO: `/v3/users/d4a608563189457dbcadf8d48acdd84a` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:48.584001 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4a608563189457dbcadf8d48acdd84a)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:48.585906 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:48.649173 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4a608563189457dbcadf8d48acdd84a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:48.656420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4a608563189457dbcadf8d48acdd84a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:48.657169 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4a608563189457dbcadf8d48acdd84a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:48.657853 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4a608563189457dbcadf8d48acdd84a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:48.670244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-404255a9-bd70-4f1f-9822-48679d29c2a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4a608563189457dbcadf8d48acdd84a'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:48.684065 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1401/2832] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:25:48 2022] DELETE /identity/v3/users/d4a608563189457dbcadf8d48acdd84a => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:25:48.854622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:48.870078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0aa805b-05a0-4175-bdfd-ca637ad171e6 service placement] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:48.927411 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:48.929116 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:48.929403 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:48.929680 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] PATH_INFO: `/v3/projects/f0c19d009f1b4ef09a40267bcf654107` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:48.930886 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0c19d009f1b4ef09a40267bcf654107)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:48.932917 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:48.978894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0c19d009f1b4ef09a40267bcf654107', 'request_id': 'req-1ea80bc4-7a7b-4058-8444-2958b5a2390c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:48.984377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0c19d009f1b4ef09a40267bcf654107', 'request_id': 'req-1ea80bc4-7a7b-4058-8444-2958b5a2390c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:48.990743 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0c19d009f1b4ef09a40267bcf654107', 'request_id': 'req-1ea80bc4-7a7b-4058-8444-2958b5a2390c'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:49.002200 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] The token cache is being invalidate because project f0c19d009f1b4ef09a40267bcf654107 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80268) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:25:49.002487 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:49.003104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-1ea80bc4-7a7b-4058-8444-2958b5a2390c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:25:49.004684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1432/2833] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:25:48 2022] DELETE /identity/v3/projects/f0c19d009f1b4ef09a40267bcf654107 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:25:49.010140 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76730f1c-8554-4c92-bdb7-9e79f7bc7d61 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.010430 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76730f1c-8554-4c92-bdb7-9e79f7bc7d61 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.010710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76730f1c-8554-4c92-bdb7-9e79f7bc7d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.026344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:25:49.059118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-76730f1c-8554-4c92-bdb7-9e79f7bc7d61 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:25:49.060321 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-76730f1c-8554-4c92-bdb7-9e79f7bc7d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:49.118711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1402/2834] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:25:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:25:49.126327 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393cdb1-2540-4aeb-8bbf-02313d1a520c None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.126622 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393cdb1-2540-4aeb-8bbf-02313d1a520c None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.126910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5393cdb1-2540-4aeb-8bbf-02313d1a520c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.145425 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-5393cdb1-2540-4aeb-8bbf-02313d1a520c None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:25:49.146628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-5393cdb1-2540-4aeb-8bbf-02313d1a520c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:49.154506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1433/2835] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:25:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:25:49.158466 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76730f1c-8554-4c92-bdb7-9e79f7bc7d61 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:49.167946 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76730f1c-8554-4c92-bdb7-9e79f7bc7d61 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:49.172489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:49.173133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.173420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.173980 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] PATH_INFO: `/v3/domains` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.174705 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80269) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:25:49.175012 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:49.176683 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:49.189046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1403/2836] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:25:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 05 10:25:49.193099 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5393cdb1-2540-4aeb-8bbf-02313d1a520c None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:49.202961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5393cdb1-2540-4aeb-8bbf-02313d1a520c None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:49.207748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:49.208416 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.208717 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.209000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] PATH_INFO: `/v3/projects` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.209839 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:49.211748 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:49.227066 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:25:49.243962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1434/2837] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:25:49 2022] POST /identity/v3/projects => generated 377 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 05 10:25:49.245136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:49.255307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d362c01-50f4-4d85-a3e8-0cacbd106f60 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:49.313350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:49.313876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.314155 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.314437 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] PATH_INFO: `/v3/users` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.315122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:49.317092 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:49.337887 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1404/2838] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:25:49 2022] POST /identity/v3/users => generated 289 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:25:49.341924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:49.352002 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db819bcb-1533-4beb-83ee-e9d18a46c24d admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:49.356579 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:49.357229 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.357512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.357784 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.358504 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:49.360098 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:49.370899 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1435/2839] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:25:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:25:49.374839 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:49.387475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a2180b-6472-4b04-abdb-309772ee8156 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:49.392107 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fcc50e-dd03-4b88-a0e0-87899f391a84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:49.392845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fcc50e-dd03-4b88-a0e0-87899f391a84 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.393132 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fcc50e-dd03-4b88-a0e0-87899f391a84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.393420 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fcc50e-dd03-4b88-a0e0-87899f391a84 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.394149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fcc50e-dd03-4b88-a0e0-87899f391a84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:49.395744 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8fcc50e-dd03-4b88-a0e0-87899f391a84 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:49.406388 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1405/2840] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:25:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:25:49.410534 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:49.420825 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e1d3ae-061d-4ecd-9c73-06f423f33fc0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:49.425437 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:49.426313 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.426598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.426931 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] PATH_INFO: `/v3/projects/8b5c262b91db42cb9e4042b180f9fb4d/users/b131286ad9144575a13972e25cab20db/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.442082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b5c262b91db42cb9e4042b180f9fb4d, user_id=b131286ad9144575a13972e25cab20db, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:49.444812 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:49.457481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1436/2841] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:25:49 2022] PUT /identity/v3/projects/8b5c262b91db42cb9e4042b180f9fb4d/users/b131286ad9144575a13972e25cab20db/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:25:49.462474 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ce630-e40f-4000-bc29-d554f4cdc3ca None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.462769 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ce630-e40f-4000-bc29-d554f4cdc3ca None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.463083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8ce630-e40f-4000-bc29-d554f4cdc3ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.481552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-ec8ce630-e40f-4000-bc29-d554f4cdc3ca None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:25:49.482765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-ec8ce630-e40f-4000-bc29-d554f4cdc3ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:49.528549 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:25:49.541098 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1406/2842] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:25:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 05 10:25:49.550604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:49.552308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:49.609252 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c0881-b18b-46b2-90be-841ec9aab6c6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:49.612915 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:49.613560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:49.613846 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:49.614127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:49.623013 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:49.624467 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:49.634067 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1437/2843] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:53.595004 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24730fbc-1903-43fb-8bf5-9000cbdade91 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:53.595520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24730fbc-1903-43fb-8bf5-9000cbdade91 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:53.595520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24730fbc-1903-43fb-8bf5-9000cbdade91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:53.623113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-24730fbc-1903-43fb-8bf5-9000cbdade91 None None] MFA Rules not processed for user `b131286ad9144575a13972e25cab20db`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:25:53.624697 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-24730fbc-1903-43fb-8bf5-9000cbdade91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:53.694316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1407/2844] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:25:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:25:53.710609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:53.712217 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:53.785663 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd6dec-6a0e-40a4-91df-52d40bf62ff0 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:53.791928 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:53.792630 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:53.792924 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:53.793203 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:53.801488 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:53.803997 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:53.813336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1438/2845] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:53 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:25:55.211876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:55.212333 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:55.212471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:55.238081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] MFA Rules not processed for user `b131286ad9144575a13972e25cab20db`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:25:55.241367 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:55.259019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1408/2846] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:25:55 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:25:59.400113 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:25:59.417320 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb40a05-5ab1-4da8-bd5a-89f71ece07f3 service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:25:59.424732 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfafa7e-2d44-455a-983c-c45ec2858d18 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:25:59.425903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfafa7e-2d44-455a-983c-c45ec2858d18 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:25:59.426435 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfafa7e-2d44-455a-983c-c45ec2858d18 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:25:59.427632 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfafa7e-2d44-455a-983c-c45ec2858d18 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:25:59.429854 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-dbfafa7e-2d44-455a-983c-c45ec2858d18 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:25:59.444723 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfafa7e-2d44-455a-983c-c45ec2858d18 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:25:59.446336 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfafa7e-2d44-455a-983c-c45ec2858d18 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:25:59.458735 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1439/2847] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:25:59 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:26:05.229895 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=37 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:26:09.454457 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=83,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:27:52.732864 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:27:52.734252 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:27:52.809693 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd725bbd-2896-4e99-b3db-f8590441f1a6 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:27:52.813354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:27:52.814007 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:27:52.814294 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:27:52.814573 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:27:52.816339 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:27:52.873817 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:27:52.875271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:27:52.883959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1409/2848] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:27:52 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:28:02.879959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=57 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:28:03.852653 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebdecfb-9623-4505-92ec-2b64d0242f1e None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:28:03.853170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebdecfb-9623-4505-92ec-2b64d0242f1e None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:28:03.853277 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebdecfb-9623-4505-92ec-2b64d0242f1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:28:03.870219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-0ebdecfb-9623-4505-92ec-2b64d0242f1e None None] MFA Rules not processed for user `c9203890ad7649f897b13ac7bdf6be3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:28:03.871401 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0ebdecfb-9623-4505-92ec-2b64d0242f1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:28:03.878830 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1440/2849] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:28:03 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:28:04.605564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:28:04.607633 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:28:04.676604 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6b5c67-dbdc-4117-b01a-3a641594be9d service cinder] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:28:04.681507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:28:04.682154 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:28:04.682436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:28:04.682712 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:28:04.684504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:28:04.695581 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:28:04.697178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:28:04.707481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1410/2850] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:28:04 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:28:07.411807 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebdecfb-9623-4505-92ec-2b64d0242f1e None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:28:07.427135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebdecfb-9623-4505-92ec-2b64d0242f1e None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:28:07.432884 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:28:07.433572 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:28:07.433912 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:28:07.434191 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:28:07.436045 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:28:07.449355 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:28:07.450946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:28:07.462541 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1441/2851] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:28:07 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:28:14.703242 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=32 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:28:17.457441 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=9 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:28:19.636423 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:28:19.637900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:28:19.655998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c871b570-980f-4b70-af64-aeb3376898be service placement] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:28:19.661411 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:28:19.662061 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:28:19.662344 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:28:19.662622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:28:19.664674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:28:19.677522 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:28:19.679091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:28:19.689713 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1411/2852] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:28:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:28:22.356087 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:28:22.365404 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf5dccc-8870-44d0-9da5-8a73ce9fc056 service nova] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:28:22.368973 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beecaded-e45d-4111-8d94-e6415dbcd5c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:28:22.369609 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecaded-e45d-4111-8d94-e6415dbcd5c6 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:28:22.369893 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecaded-e45d-4111-8d94-e6415dbcd5c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:28:22.370170 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beecaded-e45d-4111-8d94-e6415dbcd5c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:28:22.371892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-beecaded-e45d-4111-8d94-e6415dbcd5c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:28:22.433790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beecaded-e45d-4111-8d94-e6415dbcd5c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:28:22.435684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beecaded-e45d-4111-8d94-e6415dbcd5c6 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:28:22.445198 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1442/2853] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:28:22 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:28:29.685221 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=6 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:28:32.440838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=30 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:29:26.300021 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:29:26.301428 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:29:26.315122 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13864fc0-dd19-476e-b862-1ac15f2f2c77 service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:29:26.319928 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:29:26.320593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:29:26.320876 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:29:26.321151 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:29:26.322842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:29:26.333958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:29:26.335515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:29:26.345803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1412/2854] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:29:26 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:29:36.341585 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=6 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:30:20.685584 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6f273-5d3b-4487-a1da-122d511e0eba None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:20.686135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6f273-5d3b-4487-a1da-122d511e0eba None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:20.686135 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec6f273-5d3b-4487-a1da-122d511e0eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:20.703280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-dec6f273-5d3b-4487-a1da-122d511e0eba None None] MFA Rules not processed for user `b131286ad9144575a13972e25cab20db`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:30:20.704515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-dec6f273-5d3b-4487-a1da-122d511e0eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:20.711304 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1443/2855] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:30:20 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:30:30.698301 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:30:37.261390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:30:37.270821 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4eecf2-5b02-4dda-b186-da158ec17d85 service placement] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:30:37.274403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:30:37.275039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:37.275328 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:37.275605 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:37.277909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:37.294728 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:30:37.296286 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:30:37.308055 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1413/2856] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:30:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:30:38.499940 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec6f273-5d3b-4487-a1da-122d511e0eba None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:30:38.509790 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec6f273-5d3b-4487-a1da-122d511e0eba None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:30:38.513422 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:30:38.514055 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:38.514331 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:38.514604 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:38.516347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:38.529423 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:30:38.530831 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:30:38.541863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1444/2857] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:30:38 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:30:41.467720 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:30:41.469175 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:41.480057 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11ae5cc-53b7-4d46-9a19-9fdaeacbd8ec service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:30:41.483708 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:30:41.484284 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:41.484594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:41.484872 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:41.486606 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:41.497877 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:30:41.499281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:30:41.508790 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1414/2858] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:30:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:30:47.957797 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:30:47.962185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:47.995126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe7dbdf-dbd0-46f3-aac7-e92d8bd3fe7f service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:30:48.015470 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:30:48.016789 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:48.017161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:48.017627 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] PATH_INFO: `/v3/users/c9203890ad7649f897b13ac7bdf6be3b` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:48.019017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9203890ad7649f897b13ac7bdf6be3b)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:30:48.021859 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:30:48.074377 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9203890ad7649f897b13ac7bdf6be3b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.075044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9203890ad7649f897b13ac7bdf6be3b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.082292 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9203890ad7649f897b13ac7bdf6be3b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.083016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9203890ad7649f897b13ac7bdf6be3b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.091346 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9203890ad7649f897b13ac7bdf6be3b'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.116705 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1445/2859] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:30:47 2022] DELETE /identity/v3/users/c9203890ad7649f897b13ac7bdf6be3b => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:30:48.295749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:30:48.313083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840abd70-07ee-48cb-af1e-26b4da9c3f1b service neutron] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:30:48.375552 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:30:48.376451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:48.376775 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:48.377062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] PATH_INFO: `/v3/projects/86eac45249ad4a5e9513fcc091bcf42e` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:48.378542 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] RBAC: Authorizing `identity:delete_project(project_id=86eac45249ad4a5e9513fcc091bcf42e)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:30:48.380657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:30:48.432259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86eac45249ad4a5e9513fcc091bcf42e', 'request_id': 'req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.438788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86eac45249ad4a5e9513fcc091bcf42e', 'request_id': 'req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.445564 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86eac45249ad4a5e9513fcc091bcf42e', 'request_id': 'req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.459529 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] The token cache is being invalidate because project 86eac45249ad4a5e9513fcc091bcf42e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:30:48.459824 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.460530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.462139 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1415/2860] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:30:48 2022] DELETE /identity/v3/projects/86eac45249ad4a5e9513fcc091bcf42e => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:30:48.466223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:30:48.467620 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:48.481245 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e810f-cc72-401c-ad67-e620878bf517 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:30:48.540564 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:30:48.541531 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:48.541816 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:48.542234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] PATH_INFO: `/v3/users/888de2e07aca4761bed43fb8b8e5f38a` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:48.543451 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] RBAC: Authorizing `identity:delete_user(user_id=888de2e07aca4761bed43fb8b8e5f38a)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:30:48.545716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:30:48.594799 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '888de2e07aca4761bed43fb8b8e5f38a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.595472 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '888de2e07aca4761bed43fb8b8e5f38a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.602813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '888de2e07aca4761bed43fb8b8e5f38a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.603569 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '888de2e07aca4761bed43fb8b8e5f38a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.607568 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:30:48.611953 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '888de2e07aca4761bed43fb8b8e5f38a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.642561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1446/2861] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:30:48 2022] DELETE /identity/v3/users/888de2e07aca4761bed43fb8b8e5f38a => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:30:48.826765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:30:48.828236 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:30:48.844093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a469ee3f-4dfb-4104-8383-1cfdef4dcf94 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:30:48.910407 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:30:48.911280 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:30:48.911565 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:30:48.911842 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] PATH_INFO: `/v3/projects/78c9221526824231babe7a480dae7bef` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:30:48.913062 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78c9221526824231babe7a480dae7bef)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:30:48.914998 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:30:48.922951 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:30:48.970482 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78c9221526824231babe7a480dae7bef', 'request_id': 'req-dc5514b5-59b1-4852-a771-f763a6bdf633'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.976238 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78c9221526824231babe7a480dae7bef', 'request_id': 'req-dc5514b5-59b1-4852-a771-f763a6bdf633'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.984427 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78c9221526824231babe7a480dae7bef', 'request_id': 'req-dc5514b5-59b1-4852-a771-f763a6bdf633'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.996679 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] The token cache is being invalidate because project 78c9221526824231babe7a480dae7bef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:30:48.996900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.997543 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:30:48.999104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1416/2862] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:30:48 2022] DELETE /identity/v3/projects/78c9221526824231babe7a480dae7bef => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:30:58.638529 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=8,keystone:DELETE=2 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:30:58.993433 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=22,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:31:32.579257 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:31:32.580863 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:31:32.637526 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7ea898-f22d-4936-abf4-ca14ae2ba362 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:31:32.684514 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:31:32.685161 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:31:32.685442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:31:32.685718 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:31:32.687476 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:31:32.696601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:31:32.698017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:31:32.706560 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1447/2863] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:31:32 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:31:40.893788 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:31:40.895307 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:31:40.969334 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5514b5-59b1-4852-a771-f763a6bdf633 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:31:40.974822 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f99a9fe4b40646d389c3ebf47358956c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:31:40.975475 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:31:40.975765 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:31:40.976046 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:31:40.977845 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:31:41.034463 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:31:41.036054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:31:41.045519 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1417/2864] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:31:40 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:31:42.702869 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=46 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:31:51.040911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=56 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:11.345264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:11.346682 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:11.356985 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948e9770-f71e-4226-a22c-2287288832ff service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:11.360594 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:11.361223 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:11.361505 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:11.361780 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:11.363440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:11.426535 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:11.428052 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:11.438020 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1448/2865] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:14.413724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:14.415289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:14.465018 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:14.466506 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:14.478851 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2747a2-fc2e-4b2d-8061-6bf769eb3f42 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:14.483528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:14.485034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:14.485312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:14.485586 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] PATH_INFO: `/v3/users/b131286ad9144575a13972e25cab20db` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:14.486770 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] RBAC: Authorizing `identity:delete_user(user_id=b131286ad9144575a13972e25cab20db)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:14.488722 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:14.496013 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c039b-318d-42cd-8287-435a870a8249 service nova] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:14.501202 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:14.501957 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:14.502244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:14.502539 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:14.504311 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:14.516603 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:14.518429 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:14.529683 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1418/2866] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:14 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:14.538649 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b131286ad9144575a13972e25cab20db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.539222 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b131286ad9144575a13972e25cab20db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.547495 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b131286ad9144575a13972e25cab20db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.548367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b131286ad9144575a13972e25cab20db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.556003 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-f81fd5a4-58c3-44e6-bc84-12033ffde13c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b131286ad9144575a13972e25cab20db'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.574637 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1449/2867] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:32:14 2022] DELETE /identity/v3/users/b131286ad9144575a13972e25cab20db => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:32:14.744471 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:14.756762 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cbda32-2ea8-4645-beb7-6679e7bbc28f service placement] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:14.813554 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:14.814406 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:14.814686 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:14.814960 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] PATH_INFO: `/v3/projects/8b5c262b91db42cb9e4042b180f9fb4d` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:14.816118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b5c262b91db42cb9e4042b180f9fb4d)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:14.818058 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:14.862786 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b5c262b91db42cb9e4042b180f9fb4d', 'request_id': 'req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.868133 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b5c262b91db42cb9e4042b180f9fb4d', 'request_id': 'req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.874104 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b5c262b91db42cb9e4042b180f9fb4d', 'request_id': 'req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.885109 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] The token cache is being invalidate because project 8b5c262b91db42cb9e4042b180f9fb4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:32:14.885396 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.886016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11ec16e8-6446-4ea3-8ab6-cb8915a15fbc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:14.887521 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1419/2868] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:32:14 2022] DELETE /identity/v3/projects/8b5c262b91db42cb9e4042b180f9fb4d => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:14.893628 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418eeea5-acce-4936-9811-6255c16aebf1 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:14.893920 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418eeea5-acce-4936-9811-6255c16aebf1 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:14.894197 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418eeea5-acce-4936-9811-6255c16aebf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:14.912542 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-418eeea5-acce-4936-9811-6255c16aebf1 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:14.913726 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-418eeea5-acce-4936-9811-6255c16aebf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:14.972990 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1450/2869] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:14.980902 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9f33a-2333-4a8f-9dcd-12e7932e3fa8 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:14.981194 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9f33a-2333-4a8f-9dcd-12e7932e3fa8 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:14.981503 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe9f33a-2333-4a8f-9dcd-12e7932e3fa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:14.999602 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-fbe9f33a-2333-4a8f-9dcd-12e7932e3fa8 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:15.000783 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-fbe9f33a-2333-4a8f-9dcd-12e7932e3fa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:15.008610 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1420/2870] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:15.012670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418eeea5-acce-4936-9811-6255c16aebf1 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.022892 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418eeea5-acce-4936-9811-6255c16aebf1 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.028043 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.029318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.029318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.029551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.030264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:32:15.031182 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.033109 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.034747 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:15.055640 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1451/2871] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:32:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:32:15.056882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9f33a-2333-4a8f-9dcd-12e7932e3fa8 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.066244 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe9f33a-2333-4a8f-9dcd-12e7932e3fa8 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.070907 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.071560 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.071846 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.072125 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.072857 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.074756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.084808 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:15.112627 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1421/2872] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:32:15 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:15.116729 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.127000 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af4643-7465-4ef7-808b-aef79610b1f9 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.182611 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.183278 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.183566 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.183844 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.184554 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.186481 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.208163 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1452/2873] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:32:15 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:15.212271 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.222709 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1863776-b93f-4b4e-80f2-3355bb3570ce admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.227324 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.227982 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.228272 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.228588 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.229349 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.230988 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.240890 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1422/2874] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:15.244562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.255091 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911463e8-cef2-4393-ac68-3e0df5d8bfb1 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.259763 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5646feb8-7bad-4e17-b037-a1f10b5b7306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.260428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5646feb8-7bad-4e17-b037-a1f10b5b7306 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.260712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5646feb8-7bad-4e17-b037-a1f10b5b7306 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.260987 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5646feb8-7bad-4e17-b037-a1f10b5b7306 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.261712 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5646feb8-7bad-4e17-b037-a1f10b5b7306 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.263280 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5646feb8-7bad-4e17-b037-a1f10b5b7306 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.273591 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1453/2875] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:15.277544 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.287434 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d9e7be-eabe-494d-9e8c-fd6276aadf89 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.292341 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.293285 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.293572 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.293851 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] PATH_INFO: `/v3/projects/7e41343be313414b91f63aa4f5c9955e/users/c648372ba2f546548d7cb82abe2b6f87/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.308277 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e41343be313414b91f63aa4f5c9955e, user_id=c648372ba2f546548d7cb82abe2b6f87, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.310959 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.323813 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1423/2876] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:32:15 2022] PUT /identity/v3/projects/7e41343be313414b91f63aa4f5c9955e/users/c648372ba2f546548d7cb82abe2b6f87/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:32:15.335244 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6538fece-370e-4b0e-8a4b-71475c599d76 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.335537 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6538fece-370e-4b0e-8a4b-71475c599d76 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.335817 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6538fece-370e-4b0e-8a4b-71475c599d76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.361558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-6538fece-370e-4b0e-8a4b-71475c599d76 None None] MFA Rules not processed for user `c648372ba2f546548d7cb82abe2b6f87`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:15.362766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-6538fece-370e-4b0e-8a4b-71475c599d76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:15.427368 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1454/2877] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:15.431840 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.494020 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d1a0e8-ba64-45c4-960f-c8c50da422d4 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.498639 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.499841 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.500142 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.500414 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.501749 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.503054 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.530879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1424/2878] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:32:15 2022] POST /identity/v3/projects => generated 385 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:15.535062 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6538fece-370e-4b0e-8a4b-71475c599d76 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.545599 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6538fece-370e-4b0e-8a4b-71475c599d76 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.600671 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.601332 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.601615 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.601894 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.602589 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.604544 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.612445 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:15.626195 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1455/2879] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:32:15 2022] POST /identity/v3/users => generated 299 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:15.629350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.639656 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f6c165-f9e6-4365-9c89-b04b8943f466 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.644354 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.645042 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.645336 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.645621 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.646892 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.648563 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.659636 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1425/2880] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:15.663709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.674080 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfb1e58-e100-4928-b357-d605b5ad4cae admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.678678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.679555 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.679838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.680118 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] PATH_INFO: `/v3/projects/fc950e48e560463dbf1e6e0f111da3d8/users/7f91cb10a4d148c2a89b80adc6b34b4a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.695211 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc950e48e560463dbf1e6e0f111da3d8, user_id=7f91cb10a4d148c2a89b80adc6b34b4a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.697930 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.710463 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1456/2881] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:32:15 2022] PUT /identity/v3/projects/fc950e48e560463dbf1e6e0f111da3d8/users/7f91cb10a4d148c2a89b80adc6b34b4a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:15.714290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.724219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c540df86-b833-42a9-a597-79eb7c3d622b admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.780556 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f116d0-f68f-4f78-a99d-d7ff52fd0fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.781156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f116d0-f68f-4f78-a99d-d7ff52fd0fc7 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.781451 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f116d0-f68f-4f78-a99d-d7ff52fd0fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.781984 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f116d0-f68f-4f78-a99d-d7ff52fd0fc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.782711 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f116d0-f68f-4f78-a99d-d7ff52fd0fc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.784327 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f116d0-f68f-4f78-a99d-d7ff52fd0fc7 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.793994 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1426/2882] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:15 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:15.798582 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:15.808746 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724213dc-243b-420a-946e-7ca28a1a41be admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:15.813946 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:15.814822 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] REQUEST_METHOD: `PUT` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.815104 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.815379 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] PATH_INFO: `/v3/projects/fc950e48e560463dbf1e6e0f111da3d8/users/7f91cb10a4d148c2a89b80adc6b34b4a/roles/1f92a800392649b3b251840ec8dfa2e6` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.817367 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc950e48e560463dbf1e6e0f111da3d8, user_id=7f91cb10a4d148c2a89b80adc6b34b4a, role_id=1f92a800392649b3b251840ec8dfa2e6)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:15.820009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:15.830044 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1457/2883] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:32:15 2022] PUT /identity/v3/projects/fc950e48e560463dbf1e6e0f111da3d8/users/7f91cb10a4d148c2a89b80adc6b34b4a/roles/1f92a800392649b3b251840ec8dfa2e6 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:15.887363 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:15.887657 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:15.887944 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:15.916091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] MFA Rules not processed for user `7f91cb10a4d148c2a89b80adc6b34b4a`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:15.917366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:15.923219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:15.988710 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1427/2884] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:15 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:15.999194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:16.001433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:16.057709 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0e13f-c16b-440f-96ec-296916b86790 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:16.061219 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:16.062402 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:16.062684 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:16.062988 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:16.117201 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:16.118623 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:16.127933 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1458/2885] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:15 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:20.599099 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:20.600535 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:20.612773 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cf0062-78d9-488f-9699-d0a1b29cc90c None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:20.616237 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26713469-a796-4f43-b998-aabae6a11e39 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:20.616900 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26713469-a796-4f43-b998-aabae6a11e39 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:20.617187 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26713469-a796-4f43-b998-aabae6a11e39 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:20.617473 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26713469-a796-4f43-b998-aabae6a11e39 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:20.626417 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26713469-a796-4f43-b998-aabae6a11e39 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:20.627894 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26713469-a796-4f43-b998-aabae6a11e39 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:20.637487 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1428/2886] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:20 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:25.463772 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:25.525768 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a02497-e80f-490b-a25c-c16f41bbc0ad service cinder] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:25.530883 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:25.532009 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:25.532289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:25.532605 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] PATH_INFO: `/v3/users/c648372ba2f546548d7cb82abe2b6f87` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:25.533778 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c648372ba2f546548d7cb82abe2b6f87)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:25.535690 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:25.545888 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:25.580955 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c648372ba2f546548d7cb82abe2b6f87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:25.581602 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c648372ba2f546548d7cb82abe2b6f87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:25.588563 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c648372ba2f546548d7cb82abe2b6f87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:25.589187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c648372ba2f546548d7cb82abe2b6f87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:25.597264 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c648372ba2f546548d7cb82abe2b6f87'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:25.614442 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1459/2887] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:32:25 2022] DELETE /identity/v3/users/c648372ba2f546548d7cb82abe2b6f87 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:25.618884 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:25.619350 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:25.619455 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:25.637093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:25.638292 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:25.695852 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1429/2888] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:25.705914 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:25.707347 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:25.762774 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cacd28-812e-4e3a-9d9b-5a8203683da6 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:25.767064 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:25.767739 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:25.768026 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:25.768306 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:25.777215 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:25.778650 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:25.788234 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1460/2889] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:26.049041 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:26.050469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:26.065612 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5ca907-4851-4082-8e7d-001131d01286 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:26.070290 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:26.071156 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.071441 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.071760 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] PATH_INFO: `/v3/projects/7e41343be313414b91f63aa4f5c9955e` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.073050 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e41343be313414b91f63aa4f5c9955e)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:26.074962 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:26.119999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e41343be313414b91f63aa4f5c9955e', 'request_id': 'req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.126045 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e41343be313414b91f63aa4f5c9955e', 'request_id': 'req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.132039 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e41343be313414b91f63aa4f5c9955e', 'request_id': 'req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.143366 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] The token cache is being invalidate because project 7e41343be313414b91f63aa4f5c9955e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:32:26.143644 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.144259 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.145814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1430/2890] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:32:26 2022] DELETE /identity/v3/projects/7e41343be313414b91f63aa4f5c9955e => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:26.149754 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:26.159647 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67f422-403a-432b-9ce6-ef0318fa43e9 service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:26.214970 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:26.215847 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.216131 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.216433 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] PATH_INFO: `/v3/users/7f91cb10a4d148c2a89b80adc6b34b4a` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.217642 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f91cb10a4d148c2a89b80adc6b34b4a)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:26.219556 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:26.253801 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:DELETE=15,keystone:SELECT=84,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:26.267365 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f91cb10a4d148c2a89b80adc6b34b4a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.268017 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f91cb10a4d148c2a89b80adc6b34b4a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.274890 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f91cb10a4d148c2a89b80adc6b34b4a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.275515 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f91cb10a4d148c2a89b80adc6b34b4a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.282540 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-6279f6b0-ea7f-414f-9e4d-0c75abad2f11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f91cb10a4d148c2a89b80adc6b34b4a'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.300078 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1461/2891] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:32:26 2022] DELETE /identity/v3/users/7f91cb10a4d148c2a89b80adc6b34b4a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:26.561594 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:26.563119 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:26.576561 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee532e6-d64b-4b0f-8206-e6adb4aaf206 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:26.586253 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:26.634814 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:26.635671 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.635950 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.636254 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] PATH_INFO: `/v3/projects/fc950e48e560463dbf1e6e0f111da3d8` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.637456 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc950e48e560463dbf1e6e0f111da3d8)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:26.639390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:26.684911 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc950e48e560463dbf1e6e0f111da3d8', 'request_id': 'req-0693873a-6208-48b1-b9c5-13bdcdd9aeba'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.690395 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc950e48e560463dbf1e6e0f111da3d8', 'request_id': 'req-0693873a-6208-48b1-b9c5-13bdcdd9aeba'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.696470 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc950e48e560463dbf1e6e0f111da3d8', 'request_id': 'req-0693873a-6208-48b1-b9c5-13bdcdd9aeba'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.707948 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] The token cache is being invalidate because project fc950e48e560463dbf1e6e0f111da3d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:32:26.708233 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.708879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-0693873a-6208-48b1-b9c5-13bdcdd9aeba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:26.710416 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1431/2892] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:32:26 2022] DELETE /identity/v3/projects/fc950e48e560463dbf1e6e0f111da3d8 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:26.716226 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81955b-2239-4f0a-bed3-020a659c3c30 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.716733 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81955b-2239-4f0a-bed3-020a659c3c30 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.716836 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be81955b-2239-4f0a-bed3-020a659c3c30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.735480 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-be81955b-2239-4f0a-bed3-020a659c3c30 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:26.736696 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-be81955b-2239-4f0a-bed3-020a659c3c30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:26.796338 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1462/2893] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:26.804214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db313e4d-4077-4662-b1c7-82b1213e55dc None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.804551 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db313e4d-4077-4662-b1c7-82b1213e55dc None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.804831 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db313e4d-4077-4662-b1c7-82b1213e55dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.822506 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-db313e4d-4077-4662-b1c7-82b1213e55dc None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:26.823707 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-db313e4d-4077-4662-b1c7-82b1213e55dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:26.832023 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1432/2894] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:26.836159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81955b-2239-4f0a-bed3-020a659c3c30 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:26.846610 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be81955b-2239-4f0a-bed3-020a659c3c30 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:26.851205 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:26.851858 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.852141 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.852444 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.853185 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:32:26.853494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:26.855105 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:26.867962 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1463/2895] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:32:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:32:26.872283 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db313e4d-4077-4662-b1c7-82b1213e55dc None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:26.881469 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db313e4d-4077-4662-b1c7-82b1213e55dc None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:26.886093 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:26.886741 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.887028 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.887309 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.888019 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:26.890489 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:26.916763 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1433/2896] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:32:26 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:26.921028 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:26.931133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268bf5ce-7e23-459c-a943-54e725a3ebc0 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:26.986512 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:26.987159 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:26.987440 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:26.987716 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:26.988383 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:26.990308 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:27.008686 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1464/2897] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:32:26 2022] POST /identity/v3/users => generated 287 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:27.012958 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:27.023669 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6352414f-8b02-4d09-a733-94ec81a18aab admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:27.028302 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:27.028999 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.029289 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.029568 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.030893 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:27.032803 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:27.043136 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1434/2898] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:27.047014 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:27.056841 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00af3c35-f362-410b-b876-c7ac0e39f825 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:27.061466 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145d9545-701a-4ef1-a7a4-1cee9fb244ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:27.062117 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145d9545-701a-4ef1-a7a4-1cee9fb244ab admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.062396 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145d9545-701a-4ef1-a7a4-1cee9fb244ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.062670 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145d9545-701a-4ef1-a7a4-1cee9fb244ab admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.063428 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145d9545-701a-4ef1-a7a4-1cee9fb244ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:27.065037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145d9545-701a-4ef1-a7a4-1cee9fb244ab admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:27.075082 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1465/2899] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:27.079567 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:27.089520 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a685e23-ca87-4640-8d8d-3752782c5f1a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:27.094716 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:27.095593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.095882 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.096164 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] PATH_INFO: `/v3/projects/2a0cbd539cfb41d49bd9b3e952512310/users/f789d656a39a4b7e8f72f53fb9dc57c3/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.110536 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a0cbd539cfb41d49bd9b3e952512310, user_id=f789d656a39a4b7e8f72f53fb9dc57c3, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:27.113265 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:27.126178 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1435/2900] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:32:27 2022] PUT /identity/v3/projects/2a0cbd539cfb41d49bd9b3e952512310/users/f789d656a39a4b7e8f72f53fb9dc57c3/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:27.138521 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d966d1e-c05d-4d33-8102-5b192d68b271 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.138813 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d966d1e-c05d-4d33-8102-5b192d68b271 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.139119 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d966d1e-c05d-4d33-8102-5b192d68b271 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.165791 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-1d966d1e-c05d-4d33-8102-5b192d68b271 None None] MFA Rules not processed for user `f789d656a39a4b7e8f72f53fb9dc57c3`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:27.166989 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-1d966d1e-c05d-4d33-8102-5b192d68b271 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:27.171127 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:27.233011 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1466/2901] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:27 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:27.244206 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:27.245909 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:27.289827 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:27.304770 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab3f934-a541-4bec-b370-6ac31328f84a admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:27.308261 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5db440-4d5f-4e6a-ae11-88899077d0e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:27.308932 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5db440-4d5f-4e6a-ae11-88899077d0e9 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.309214 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5db440-4d5f-4e6a-ae11-88899077d0e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.309494 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5db440-4d5f-4e6a-ae11-88899077d0e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.318113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5db440-4d5f-4e6a-ae11-88899077d0e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:27.319530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5db440-4d5f-4e6a-ae11-88899077d0e9 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:27.328816 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1436/2902] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:27 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:27.348483 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d966d1e-c05d-4d33-8102-5b192d68b271 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:27.408607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d966d1e-c05d-4d33-8102-5b192d68b271 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:27.413075 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:27.413934 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.414216 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.414490 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] PATH_INFO: `/v3/users/f789d656a39a4b7e8f72f53fb9dc57c3` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.415658 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f789d656a39a4b7e8f72f53fb9dc57c3)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:27.417598 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:27.463972 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f789d656a39a4b7e8f72f53fb9dc57c3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:27.464731 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f789d656a39a4b7e8f72f53fb9dc57c3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:27.471565 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f789d656a39a4b7e8f72f53fb9dc57c3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:27.472194 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f789d656a39a4b7e8f72f53fb9dc57c3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:27.479186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f789d656a39a4b7e8f72f53fb9dc57c3'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:27.496016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1467/2903] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:32:27 2022] DELETE /identity/v3/users/f789d656a39a4b7e8f72f53fb9dc57c3 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:27.501141 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.501436 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.501724 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.519403 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:27.520622 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:27.578806 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1437/2904] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:27.588318 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:27.589766 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:27.617558 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:27.647685 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b545f2-1c22-471d-b8e4-0a33a1428765 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:27.651241 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965d9f1e-85b4-45da-970a-8e97cc6941a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:27.651876 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965d9f1e-85b4-45da-970a-8e97cc6941a5 service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.652154 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965d9f1e-85b4-45da-970a-8e97cc6941a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.652518 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965d9f1e-85b4-45da-970a-8e97cc6941a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.661561 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965d9f1e-85b4-45da-970a-8e97cc6941a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:27.662978 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965d9f1e-85b4-45da-970a-8e97cc6941a5 service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:27.672662 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1468/2905] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:27.933235 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:27.934674 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:27.946638 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c029cc9-1e9d-45ab-82f5-673753e39894 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:27.951504 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:27.952526 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:27.952879 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:27.953226 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] PATH_INFO: `/v3/projects/2a0cbd539cfb41d49bd9b3e952512310` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:27.954500 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a0cbd539cfb41d49bd9b3e952512310)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:27.956933 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.128081 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a0cbd539cfb41d49bd9b3e952512310', 'request_id': 'req-11d799a2-64f3-42e7-a2a4-76fbf88dba92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:28.133390 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a0cbd539cfb41d49bd9b3e952512310', 'request_id': 'req-11d799a2-64f3-42e7-a2a4-76fbf88dba92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:28.139593 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a0cbd539cfb41d49bd9b3e952512310', 'request_id': 'req-11d799a2-64f3-42e7-a2a4-76fbf88dba92'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:28.151083 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] The token cache is being invalidate because project 2a0cbd539cfb41d49bd9b3e952512310 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:32:28.151365 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:28.151972 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-11d799a2-64f3-42e7-a2a4-76fbf88dba92 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:28.153530 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1438/2906] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:32:27 2022] DELETE /identity/v3/projects/2a0cbd539cfb41d49bd9b3e952512310 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:28.159757 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab73cae-fae8-4917-8255-18e091ddb847 None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.160253 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab73cae-fae8-4917-8255-18e091ddb847 None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.160385 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab73cae-fae8-4917-8255-18e091ddb847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.178607 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-aab73cae-fae8-4917-8255-18e091ddb847 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:28.179810 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-aab73cae-fae8-4917-8255-18e091ddb847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:28.242908 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1469/2907] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:28.250541 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68bfe76-37cd-4d7c-96b6-376f7a2d5537 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.250837 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68bfe76-37cd-4d7c-96b6-376f7a2d5537 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.251118 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68bfe76-37cd-4d7c-96b6-376f7a2d5537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.269149 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-f68bfe76-37cd-4d7c-96b6-376f7a2d5537 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:28.270356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-f68bfe76-37cd-4d7c-96b6-376f7a2d5537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:28.278094 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1439/2908] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:28.282877 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab73cae-fae8-4917-8255-18e091ddb847 None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:28.292494 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab73cae-fae8-4917-8255-18e091ddb847 None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:28.297016 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:28.297660 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.297944 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.298225 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.298952 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80268) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 05 10:32:28.299258 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:28.300903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.312208 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1470/2909] 10.222.0.20 () {58 vars in 1152 bytes} [Tue Jul 5 10:32:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 05 10:32:28.316247 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68bfe76-37cd-4d7c-96b6-376f7a2d5537 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:28.325347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68bfe76-37cd-4d7c-96b6-376f7a2d5537 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:28.330449 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:28.331089 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.331371 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.331648 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] PATH_INFO: `/v3/projects` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.332323 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:28.334281 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.358995 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1440/2910] 10.222.0.20 () {60 vars in 1152 bytes} [Tue Jul 5 10:32:28 2022] POST /identity/v3/projects => generated 391 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:28.363399 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:28.373492 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fb5bc0-1ba9-4260-8db1-3e7f536ca0e8 admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:28.428212 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:28.428903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.429187 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.429465 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] PATH_INFO: `/v3/users` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.430133 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:28.432034 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.450875 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1471/2911] 10.222.0.20 () {60 vars in 1143 bytes} [Tue Jul 5 10:32:28 2022] POST /identity/v3/users => generated 296 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 05 10:32:28.455555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:28.466143 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6761be1b-46a4-474a-b697-03552d133981 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:28.471375 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:28.472035 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.472316 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.472629 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] PATH_INFO: `/v3/roles` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.473356 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:28.474940 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.485219 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1441/2912] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:28.488949 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:28.498677 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6503667b-5266-4897-90cf-839a3577fe8f admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:28.503305 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3339ca-c31c-430b-8c0c-83093381b780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:28.503960 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3339ca-c31c-430b-8c0c-83093381b780 admin admin] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.504243 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3339ca-c31c-430b-8c0c-83093381b780 admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.504557 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3339ca-c31c-430b-8c0c-83093381b780 admin admin] PATH_INFO: `/v3/roles` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.505289 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3339ca-c31c-430b-8c0c-83093381b780 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:28.506872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3339ca-c31c-430b-8c0c-83093381b780 admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.517166 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1472/2913] 10.222.0.20 () {58 vars in 1121 bytes} [Tue Jul 5 10:32:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 05 10:32:28.521111 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:28.531714 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1994eb6-0470-4d22-9b32-7f6b58e78991 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:28.536373 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:28.537263 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.537545 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.537823 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] PATH_INFO: `/v3/projects/6d00ec0650fc47b3aa153f786dce08b2/users/c1c2032733b34d098f1b4ff4421f41d7/roles/eefd36f11cba4b2eac18767b1b872d97` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.552084 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d00ec0650fc47b3aa153f786dce08b2, user_id=c1c2032733b34d098f1b4ff4421f41d7, role_id=eefd36f11cba4b2eac18767b1b872d97)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:28.554756 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.567347 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1442/2914] 10.222.0.20 () {60 vars in 1482 bytes} [Tue Jul 5 10:32:28 2022] PUT /identity/v3/projects/6d00ec0650fc47b3aa153f786dce08b2/users/c1c2032733b34d098f1b4ff4421f41d7/roles/eefd36f11cba4b2eac18767b1b872d97 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 05 10:32:28.577956 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9929053-aa69-4ae5-a224-8aca99ec4fea None None] REQUEST_METHOD: `POST` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.578247 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9929053-aa69-4ae5-a224-8aca99ec4fea None None] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.578528 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9929053-aa69-4ae5-a224-8aca99ec4fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.604874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.auth.core [None req-b9929053-aa69-4ae5-a224-8aca99ec4fea None None] MFA Rules not processed for user `c1c2032733b34d098f1b4ff4421f41d7`. Rule list: `[]` (Enabled: `True`). {{(pid=80268) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:28.606089 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-b9929053-aa69-4ae5-a224-8aca99ec4fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:28.625872 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:28.672957 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1473/2915] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:28 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:28.683730 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:28.685322 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:28.691091 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:28.743715 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a29fce-7629-46c2-a2bc-6a1a9b8f07d9 admin admin] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:28.747262 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac0482e-cc64-4812-9b06-57b7855bee43 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1fc5b3da67de48cb83848168b16dd153', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:28.747898 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0482e-cc64-4812-9b06-57b7855bee43 service cinder] REQUEST_METHOD: `GET` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:28.748179 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0482e-cc64-4812-9b06-57b7855bee43 service cinder] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:28.748480 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac0482e-cc64-4812-9b06-57b7855bee43 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:28.757113 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac0482e-cc64-4812-9b06-57b7855bee43 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:28.758515 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac0482e-cc64-4812-9b06-57b7855bee43 service cinder] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:28.767608 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1443/2916] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:28 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:32:38.669998 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=17 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:38.763101 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=25 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:32:49.983903 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9929053-aa69-4ae5-a224-8aca99ec4fea None None] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:50.046235 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9929053-aa69-4ae5-a224-8aca99ec4fea None None] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:50.051175 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:50.052029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:50.052310 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:50.052678 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] PATH_INFO: `/v3/users/c1c2032733b34d098f1b4ff4421f41d7` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:50.053921 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1c2032733b34d098f1b4ff4421f41d7)` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:50.055874 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:50.099312 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1c2032733b34d098f1b4ff4421f41d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.099961 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1c2032733b34d098f1b4ff4421f41d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.106814 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1c2032733b34d098f1b4ff4421f41d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.107438 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1c2032733b34d098f1b4ff4421f41d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.115139 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.notifications [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1c2032733b34d098f1b4ff4421f41d7'} {{(pid=80268) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.132443 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1474/2917] 10.222.0.20 () {58 vars in 1223 bytes} [Tue Jul 5 10:32:49 2022] DELETE /identity/v3/users/c1c2032733b34d098f1b4ff4421f41d7 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:32:50.136934 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] REQUEST_METHOD: `POST` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:50.137440 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:50.137555 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:50.154592 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.auth.core [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] MFA Rules not processed for user `bb9174aa3ac54d708fcd3d9dc7b9b827`. Rule list: `[]` (Enabled: `True`). {{(pid=80269) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 05 10:32:50.155778 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:50.211673 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1444/2918] 10.222.0.20 () {58 vars in 957 bytes} [Tue Jul 5 10:32:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 05 10:32:50.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:50.222693 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:50.280964 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88be4a88-c1de-4324-a982-67b55ee0af8e admin admin] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:50.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36ce9bd8ce6448b8b09fe9eb7b8f7c54', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:50.286037 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:50.286551 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:50.286835 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:50.295592 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:50.297186 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:50.307029 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1475/2919] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:32:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 05 10:32:50.571453 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] Authenticating user token {{(pid=80269) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:32:50.573016 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.fernet_utils [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80269) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:32:50.585507 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1642161-aca4-422d-b09c-cf59edf42ac3 None None] Validating token access rules against request {{(pid=80269) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:32:50.590079 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb9174aa3ac54d708fcd3d9dc7b9b827', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ba1e700aae1b46039e859b8f609bb931', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80269) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:32:50.590926 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:32:50.591207 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] SCRIPT_NAME: `/identity` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:32:50.591481 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] PATH_INFO: `/v3/projects/6d00ec0650fc47b3aa153f786dce08b2` {{(pid=80269) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:32:50.592682 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d00ec0650fc47b3aa153f786dce08b2)` {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:32:50.594582 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] RBAC: Authorization granted {{(pid=80269) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:32:50.641250 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d00ec0650fc47b3aa153f786dce08b2', 'request_id': 'req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.647124 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d00ec0650fc47b3aa153f786dce08b2', 'request_id': 'req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.653231 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d00ec0650fc47b3aa153f786dce08b2', 'request_id': 'req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b'} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.666049 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] The token cache is being invalidate because project 6d00ec0650fc47b3aa153f786dce08b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80269) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 05 10:32:50.666360 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.666989 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG keystone.notifications [None req-72999cb4-7c59-4ae5-b8fe-f9b5c7e5f56b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80269) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 05 10:32:50.668532 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: [pid: 80269|app: 0|req: 1445/2920] 10.222.0.20 () {58 vars in 1232 bytes} [Tue Jul 5 10:32:50 2022] DELETE /identity/v3/projects/6d00ec0650fc47b3aa153f786dce08b2 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 05 10:33:00.302910 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:33:00.663053 dv-cinder-master-3-iscsi devstack@keystone.service[80269]: DEBUG dbcounter [-] [80269] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80269) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 05 10:34:33.223142 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] Authenticating user token {{(pid=80268) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 05 10:34:33.224601 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:34:33.360299 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c314014f-41fb-4f9b-9ac3-7589554934fb service neutron] Validating token access rules against request {{(pid=80268) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 05 10:34:33.418828 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5704f4-3e1b-44f0-bca4-117b25a8bee9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '947da12bc8304314b5e4a9768b083a7b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9d922cfbef6241f2a2ac0e51e1d62e82', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'service', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80268) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 05 10:34:33.419462 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5704f4-3e1b-44f0-bca4-117b25a8bee9 service placement] REQUEST_METHOD: `GET` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 05 10:34:33.419750 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5704f4-3e1b-44f0-bca4-117b25a8bee9 service placement] SCRIPT_NAME: `/identity` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 05 10:34:33.420036 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5704f4-3e1b-44f0-bca4-117b25a8bee9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80268) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 05 10:34:33.421838 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.fernet_utils [None req-df5704f4-3e1b-44f0-bca4-117b25a8bee9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80268) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 05 10:34:33.433562 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5704f4-3e1b-44f0-bca4-117b25a8bee9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 05 10:34:33.435126 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5704f4-3e1b-44f0-bca4-117b25a8bee9 service placement] RBAC: Authorization granted {{(pid=80268) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 05 10:34:33.445525 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: [pid: 80268|app: 0|req: 1476/2921] 10.222.0.20 () {60 vars in 1365 bytes} [Tue Jul 5 10:34:33 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 05 10:34:43.441329 dv-cinder-master-3-iscsi devstack@keystone.service[80268]: DEBUG dbcounter [-] [80268] Writing DB stats keystone:SELECT=89 {{(pid=80268) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}